Incidents reported on status page for Buttondown https://status.buttondown.com/ en https://status.buttondown.com/ Mon, 30 Jun 2025 12:12:51 +0000 https://status.buttondown.com/#3d21bafd03afa3f518fb30afdbf2f34b88498ecb83a3bf471698266173a03e84 buttondown.com recovered https://status.buttondown.com/ Mon, 30 Jun 2025 12:09:50 +0000 https://status.buttondown.com/#3d21bafd03afa3f518fb30afdbf2f34b88498ecb83a3bf471698266173a03e84 buttondown.com went down https://status.buttondown.com/ Sun, 29 Jun 2025 14:20:58 +0000 https://status.buttondown.com/#e3853f70b50b4becddc2ad66317a16867053b960dcb59b76ff8f05c03b816b64 p95 response time recovered https://status.buttondown.com/ Sun, 29 Jun 2025 13:35:54 +0000 https://status.buttondown.com/#e3853f70b50b4becddc2ad66317a16867053b960dcb59b76ff8f05c03b816b64 p95 response time went down https://status.buttondown.com/ Sun, 29 Jun 2025 13:21:00 +0000 https://status.buttondown.com/#889677d8782cdb8e332574c1897dec41859ee2851852e68ef534f5517124b2cb p95 response time recovered https://status.buttondown.com/ Sun, 29 Jun 2025 13:13:03 +0000 https://status.buttondown.com/#80035ce13e65a35570e74145e87add143ac6d1d24616037706a5ac0de0c0be4d buttondown.com/applied-cartography recovered https://status.buttondown.com/ Sun, 29 Jun 2025 12:05:49 +0000 https://status.buttondown.com/#889677d8782cdb8e332574c1897dec41859ee2851852e68ef534f5517124b2cb p95 response time went down https://status.buttondown.com/ Sun, 29 Jun 2025 12:05:01 +0000 https://status.buttondown.com/#80035ce13e65a35570e74145e87add143ac6d1d24616037706a5ac0de0c0be4d buttondown.com/applied-cartography went down https://status.buttondown.com/ Fri, 27 Jun 2025 19:58:55 +0000 https://status.buttondown.com/#35a1af5743a29aa76a0be182e5ff979daaaf8023115a786752cb9bbeaa1481cf buttondown.com recovered https://status.buttondown.com/ Fri, 27 Jun 2025 19:43:54 +0000 https://status.buttondown.com/#35a1af5743a29aa76a0be182e5ff979daaaf8023115a786752cb9bbeaa1481cf buttondown.com went down https://status.buttondown.com/ Fri, 27 Jun 2025 19:37:59 +0000 https://status.buttondown.com/#7ac12c8ac925e7b4a2f3e37b0bf6406a0313a2b34abffb44327613e7dcfeee9b buttondown.com recovered https://status.buttondown.com/ Fri, 27 Jun 2025 19:31:53 +0000 https://status.buttondown.com/#7ac12c8ac925e7b4a2f3e37b0bf6406a0313a2b34abffb44327613e7dcfeee9b buttondown.com went down https://status.buttondown.com/ Thu, 26 Jun 2025 18:00:43 +0000 https://status.buttondown.com/#2581f7cbb842c149c9c0d0df5fd2989a801b9ba2d0ba953a4e48d3ac7cd171c9 p95 response time recovered https://status.buttondown.com/ Thu, 26 Jun 2025 17:45:34 +0000 https://status.buttondown.com/#2581f7cbb842c149c9c0d0df5fd2989a801b9ba2d0ba953a4e48d3ac7cd171c9 p95 response time went down https://status.buttondown.com/incident/607839 Mon, 23 Jun 2025 18:45:00 -0000 https://status.buttondown.com/incident/607839#29cc2cacaf77e405a152885eb4093083a30a07a1b9516cdf5a9f605e1f9b973e We've identified that our hosting provider was incorrectly identifying incoming traffic as being from bots. This issue is now resolved, and all services have been restored. https://status.buttondown.com/incident/607839 Mon, 23 Jun 2025 18:45:00 -0000 https://status.buttondown.com/incident/607839#29cc2cacaf77e405a152885eb4093083a30a07a1b9516cdf5a9f605e1f9b973e We've identified that our hosting provider was incorrectly identifying incoming traffic as being from bots. This issue is now resolved, and all services have been restored. https://status.buttondown.com/ Mon, 23 Jun 2025 18:41:49 +0000 https://status.buttondown.com/#9e5a9160149dbba8a4eccc8c4f43f202ad1da5fb04e2a4d38b511fdefd5fa3d0 buttondown.com recovered https://status.buttondown.com/ Mon, 23 Jun 2025 18:26:50 +0000 https://status.buttondown.com/#9e5a9160149dbba8a4eccc8c4f43f202ad1da5fb04e2a4d38b511fdefd5fa3d0 buttondown.com went down https://status.buttondown.com/incident/607839 Mon, 23 Jun 2025 18:25:00 -0000 https://status.buttondown.com/incident/607839#79ed7de228dac5f8a4c0f9b789320d5658dc4971b183b7479809e9d26b4c6b0a We're investigating reports of Vercel error messages on our marketing site, documentation site, and while logging into the Buttondown app. The API, and the outbound email backlog are *not* impacted by this incident. https://status.buttondown.com/incident/607839 Mon, 23 Jun 2025 18:25:00 -0000 https://status.buttondown.com/incident/607839#79ed7de228dac5f8a4c0f9b789320d5658dc4971b183b7479809e9d26b4c6b0a We're investigating reports of Vercel error messages on our marketing site, documentation site, and while logging into the Buttondown app. The API, and the outbound email backlog are *not* impacted by this incident. https://status.buttondown.com/ Mon, 23 Jun 2025 12:21:15 +0000 https://status.buttondown.com/#6b56113aef298a9af4f14e3b01bd3ef308a4822006750099f89a81fd2d8e7c00 buttondown.com/applied-cartography recovered https://status.buttondown.com/ Mon, 23 Jun 2025 12:13:11 +0000 https://status.buttondown.com/#6b56113aef298a9af4f14e3b01bd3ef308a4822006750099f89a81fd2d8e7c00 buttondown.com/applied-cartography went down https://status.buttondown.com/ Sat, 21 Jun 2025 14:50:44 +0000 https://status.buttondown.com/#22fb28fc4bf6a86ee52fea3a94bb07614d99d27ca028400c977794f84558eb81 External event backlog recovered https://status.buttondown.com/ Sat, 21 Jun 2025 14:30:44 +0000 https://status.buttondown.com/#22fb28fc4bf6a86ee52fea3a94bb07614d99d27ca028400c977794f84558eb81 External event backlog went down https://status.buttondown.com/ Tue, 17 Jun 2025 15:40:37 +0000 https://status.buttondown.com/#30fd71cc33cc4b774f85a67676a9ac0d27ff3574b31e7e620ae0fe690f3cd08f External event backlog recovered https://status.buttondown.com/ Tue, 17 Jun 2025 15:20:40 +0000 https://status.buttondown.com/#30fd71cc33cc4b774f85a67676a9ac0d27ff3574b31e7e620ae0fe690f3cd08f External event backlog went down https://status.buttondown.com/ Mon, 16 Jun 2025 13:30:31 +0000 https://status.buttondown.com/#7d97f2cd0f87c6cbf71b4967023282c2903154d461bc77b26fc3f93444c7e76d p95 response time recovered https://status.buttondown.com/ Mon, 16 Jun 2025 13:15:32 +0000 https://status.buttondown.com/#7d97f2cd0f87c6cbf71b4967023282c2903154d461bc77b26fc3f93444c7e76d p95 response time went down https://status.buttondown.com/ Fri, 13 Jun 2025 13:30:45 +0000 https://status.buttondown.com/#7184bb9aab4cafb1e1f623fd51819c4180a25efbf7fa023a32d5b2020c5f1f3b External event backlog recovered https://status.buttondown.com/ Fri, 13 Jun 2025 13:20:45 +0000 https://status.buttondown.com/#7184bb9aab4cafb1e1f623fd51819c4180a25efbf7fa023a32d5b2020c5f1f3b External event backlog went down https://status.buttondown.com/incident/600179 Wed, 11 Jun 2025 05:30:00 -0000 https://status.buttondown.com/incident/600179#9d8199bd755964dddfef56f87f2b7d5806168efa693502fe3259c47409a53fae The incident is now resolved, and [Heroku](https://status.heroku.com/incidents/2822) is back online. (HugOps to their team!) All services are operational, and Buttondown emails are being sent without delay. https://status.buttondown.com/ Wed, 11 Jun 2025 01:20:51 +0000 https://status.buttondown.com/#a04af242de4b32aa31338e62c64d99d4bb282a22934996da4aa8843b88fcaf16 External event backlog recovered https://status.buttondown.com/ Wed, 11 Jun 2025 01:00:56 +0000 https://status.buttondown.com/#a04af242de4b32aa31338e62c64d99d4bb282a22934996da4aa8843b88fcaf16 External event backlog went down https://status.buttondown.com/incident/600179 Tue, 10 Jun 2025 22:00:00 -0000 https://status.buttondown.com/incident/600179#86c93dd0ea9349806db5bd0c400fcff492d45a86012eba2ecf76d94bed421034 We're seeing improvements for this ongoing incident: - Previously delayed emails have now been sent - Our backlog is clearing in a timely manner - New custom hosting domains are connecting as expected But as Heroku is [still reporting this as an active outage](https://status.heroku.com/incidents/2822), we're continuing to monitor. https://status.buttondown.com/ Tue, 10 Jun 2025 19:30:56 +0000 https://status.buttondown.com/#2ab8523d1faec53a02284c008b4565ff30aad41967d1e0f6d69966e67b52de33 External event backlog recovered https://status.buttondown.com/ Tue, 10 Jun 2025 18:50:56 +0000 https://status.buttondown.com/#2ab8523d1faec53a02284c008b4565ff30aad41967d1e0f6d69966e67b52de33 External event backlog went down https://status.buttondown.com/incident/600179 Tue, 10 Jun 2025 14:32:00 -0000 https://status.buttondown.com/incident/600179#de7febb84dbfb21ab7e5a25bb9402b6bf28ffb71ad0e8f8bf854a37b17c09a90 Our upstream hosting provider, Heroku, is experiencing an [ongoing outage](https://status.heroku.com/incidents/2822). As a result: - We are unable to scale up our servers to match fluctuations in the number of emails our system is sending. **This means that emails may sit in queue and take longer to send out — but they will eventually send without any intervention necessary.** - We are unable to register new custom domains for archives. All other areas of Buttondown are **unaffected**. - Emails **will eventually send without any intervention**, but may take longer to do so. **If you're waiting on an email to go out, please do not attempt to send it a second time.** - Custom domains that are already set up will work as usual. - Custom domain setup for archives will work again [after Heroku's incident is closed](https://status.heroku.com/incidents/2822). https://status.buttondown.com/ Mon, 09 Jun 2025 16:02:56 +0000 https://status.buttondown.com/#766de8993ac4f150b930be6c5b9d844d3b1512feb26d328411666aa1112dcb02 buttondown.com/applied-cartography recovered https://status.buttondown.com/ Mon, 09 Jun 2025 15:56:56 +0000 https://status.buttondown.com/#766de8993ac4f150b930be6c5b9d844d3b1512feb26d328411666aa1112dcb02 buttondown.com/applied-cartography went down https://status.buttondown.com/ Sat, 07 Jun 2025 19:10:44 +0000 https://status.buttondown.com/#1af65af92fb7b016ceed498f2189f3ecf627f34cf1178f638a873e856000010c External event backlog recovered https://status.buttondown.com/ Sat, 07 Jun 2025 19:00:35 +0000 https://status.buttondown.com/#1af65af92fb7b016ceed498f2189f3ecf627f34cf1178f638a873e856000010c External event backlog went down https://status.buttondown.com/ Wed, 04 Jun 2025 04:10:50 +0000 https://status.buttondown.com/#c4661b1734fb8f5e5413116a227f006e550d054d9735d6684419b3fcfbbf8b8e External event backlog recovered https://status.buttondown.com/ Wed, 04 Jun 2025 04:00:55 +0000 https://status.buttondown.com/#1b1a5b9bc0cdae5a0e6339e55184db55a7bd072926a21ba0ef509eacfb442f0b Backlog recovered https://status.buttondown.com/ Wed, 04 Jun 2025 03:40:55 +0000 https://status.buttondown.com/#1b1a5b9bc0cdae5a0e6339e55184db55a7bd072926a21ba0ef509eacfb442f0b Backlog went down https://status.buttondown.com/ Wed, 04 Jun 2025 03:40:52 +0000 https://status.buttondown.com/#c4661b1734fb8f5e5413116a227f006e550d054d9735d6684419b3fcfbbf8b8e External event backlog went down https://status.buttondown.com/ Fri, 30 May 2025 11:01:43 +0000 https://status.buttondown.com/#3fa571856f987077b6454eb0d5e3a9ec3619d19b79842087223afa49b99f4a3a buttondown.com recovered https://status.buttondown.com/ Fri, 30 May 2025 10:59:38 +0000 https://status.buttondown.com/#74653def0715752305e6801780433237c1deccb02c73417c5831064424c7ca01 buttondown.com/applied-cartography recovered https://status.buttondown.com/ Fri, 30 May 2025 10:59:08 +0000 https://status.buttondown.com/#3fa571856f987077b6454eb0d5e3a9ec3619d19b79842087223afa49b99f4a3a buttondown.com went down https://status.buttondown.com/ Fri, 30 May 2025 10:54:32 +0000 https://status.buttondown.com/#74653def0715752305e6801780433237c1deccb02c73417c5831064424c7ca01 buttondown.com/applied-cartography went down https://status.buttondown.com/ Fri, 30 May 2025 00:40:41 +0000 https://status.buttondown.com/#d6eb42920b8436a13795e9578c78d2143e294c5750670defba2edc055fb3b864 External event backlog recovered https://status.buttondown.com/ Fri, 30 May 2025 00:30:41 +0000 https://status.buttondown.com/#d6eb42920b8436a13795e9578c78d2143e294c5750670defba2edc055fb3b864 External event backlog went down https://status.buttondown.com/ Thu, 29 May 2025 23:10:42 +0000 https://status.buttondown.com/#8362b80e5b2e8685865dfbc50ad1209a785ec4f19104c17edb77ded3b2ed97d8 External event backlog recovered https://status.buttondown.com/ Thu, 29 May 2025 23:00:50 +0000 https://status.buttondown.com/#8362b80e5b2e8685865dfbc50ad1209a785ec4f19104c17edb77ded3b2ed97d8 External event backlog went down https://status.buttondown.com/ Sat, 24 May 2025 17:20:52 +0000 https://status.buttondown.com/#0947873c9cd30cec4abfcd2008dcfeed5d9ebf0d2c85bac91f678ea9435ba620 External event backlog recovered https://status.buttondown.com/ Sat, 24 May 2025 17:10:58 +0000 https://status.buttondown.com/#0947873c9cd30cec4abfcd2008dcfeed5d9ebf0d2c85bac91f678ea9435ba620 External event backlog went down https://status.buttondown.com/ Sat, 24 May 2025 11:05:21 +0000 https://status.buttondown.com/#6968ba7655c4dd39469f2867fdd9be0e35b150a8daad3210758c473d2d0a5a45 buttondown.com/applied-cartography recovered https://status.buttondown.com/ Sat, 24 May 2025 11:03:05 +0000 https://status.buttondown.com/#6968ba7655c4dd39469f2867fdd9be0e35b150a8daad3210758c473d2d0a5a45 buttondown.com/applied-cartography went down https://status.buttondown.com/ Thu, 22 May 2025 20:13:05 +0000 https://status.buttondown.com/#037c49cb31bf43e09623c5a21ab79da4426d0041f619bb9fb0b337d6a9eae0dc buttondown.com/applied-cartography recovered https://status.buttondown.com/ Thu, 22 May 2025 20:11:04 +0000 https://status.buttondown.com/#037c49cb31bf43e09623c5a21ab79da4426d0041f619bb9fb0b337d6a9eae0dc buttondown.com/applied-cartography went down https://status.buttondown.com/ Mon, 19 May 2025 14:25:51 +0000 https://status.buttondown.com/#df2880754228e2a51f58f216934b835dfb8fdb847ede64efba17886b1f2cd661 p95 response time recovered https://status.buttondown.com/ Mon, 19 May 2025 14:15:48 +0000 https://status.buttondown.com/#df2880754228e2a51f58f216934b835dfb8fdb847ede64efba17886b1f2cd661 p95 response time went down https://status.buttondown.com/ Wed, 14 May 2025 12:10:36 +0000 https://status.buttondown.com/#48216837659fb73930f86bd436143037bd01a910d61ff1f7fb8a6441b9238b24 External event backlog recovered https://status.buttondown.com/ Wed, 14 May 2025 12:01:03 +0000 https://status.buttondown.com/#48216837659fb73930f86bd436143037bd01a910d61ff1f7fb8a6441b9238b24 External event backlog went down https://status.buttondown.com/ Wed, 14 May 2025 10:40:41 +0000 https://status.buttondown.com/#2a2b7f543d82e166e885dbd48a96fbe29b5ac76424712fc667efb59496d02450 External event backlog recovered https://status.buttondown.com/ Wed, 14 May 2025 10:30:40 +0000 https://status.buttondown.com/#2a2b7f543d82e166e885dbd48a96fbe29b5ac76424712fc667efb59496d02450 External event backlog went down https://status.buttondown.com/ Wed, 14 May 2025 05:20:41 +0000 https://status.buttondown.com/#d5c1e1bdfe1d8d0178936839d3677356cc2cacc4b63bcbcfb711ba5b840581f4 External event backlog recovered https://status.buttondown.com/ Wed, 14 May 2025 05:10:42 +0000 https://status.buttondown.com/#d5c1e1bdfe1d8d0178936839d3677356cc2cacc4b63bcbcfb711ba5b840581f4 External event backlog went down https://status.buttondown.com/ Wed, 14 May 2025 01:00:42 +0000 https://status.buttondown.com/#685a32f7abd0e8b65cf9be8e92fbe90f261c631583ab3f72591a1b719b2cb4ef External event backlog recovered https://status.buttondown.com/ Wed, 14 May 2025 00:40:42 +0000 https://status.buttondown.com/#685a32f7abd0e8b65cf9be8e92fbe90f261c631583ab3f72591a1b719b2cb4ef External event backlog went down https://status.buttondown.com/ Mon, 12 May 2025 19:30:33 +0000 https://status.buttondown.com/#75aeca5fa95474457d9e3226b3f9060a3365c653bec44537aa2daa68fa09f697 p95 response time recovered https://status.buttondown.com/ Mon, 12 May 2025 19:15:32 +0000 https://status.buttondown.com/#75aeca5fa95474457d9e3226b3f9060a3365c653bec44537aa2daa68fa09f697 p95 response time went down https://status.buttondown.com/ Thu, 08 May 2025 03:59:48 +0000 https://status.buttondown.com/#e30e20d5115e8e9bef1f34d7dac4179d0020f078a9c76913b43255b1e33494aa buttondown.com/applied-cartography recovered https://status.buttondown.com/ Thu, 08 May 2025 03:54:41 +0000 https://status.buttondown.com/#e30e20d5115e8e9bef1f34d7dac4179d0020f078a9c76913b43255b1e33494aa buttondown.com/applied-cartography went down https://status.buttondown.com/ Thu, 01 May 2025 01:04:21 +0000 https://status.buttondown.com/#b791d7e4552da76b663a7e25749b069891095fb354e5cd421ca9eed1b9e96ff9 buttondown.com/applied-cartography recovered https://status.buttondown.com/ Thu, 01 May 2025 01:01:22 +0000 https://status.buttondown.com/#b791d7e4552da76b663a7e25749b069891095fb354e5cd421ca9eed1b9e96ff9 buttondown.com/applied-cartography went down https://status.buttondown.com/ Wed, 30 Apr 2025 08:30:55 +0000 https://status.buttondown.com/#d118ccf07ae46e4d48a320b01c02f350eee7e48bc8b6b6c26612f47956d8d283 External event backlog recovered https://status.buttondown.com/ Wed, 30 Apr 2025 08:10:59 +0000 https://status.buttondown.com/#d118ccf07ae46e4d48a320b01c02f350eee7e48bc8b6b6c26612f47956d8d283 External event backlog went down https://status.buttondown.com/ Wed, 30 Apr 2025 05:15:55 +0000 https://status.buttondown.com/#c875b5e6f270648dcef6512eec61bb194b6225482bc7a971b01fdff86cb8ee1f p95 response time recovered https://status.buttondown.com/ Wed, 30 Apr 2025 05:05:54 +0000 https://status.buttondown.com/#c875b5e6f270648dcef6512eec61bb194b6225482bc7a971b01fdff86cb8ee1f p95 response time went down https://status.buttondown.com/ Wed, 30 Apr 2025 03:25:50 +0000 https://status.buttondown.com/#691e5085d0052ad45b1a4dd0c7fbb5ae69988c9398f7e407ad4ca9f778f77eee p95 response time recovered https://status.buttondown.com/ Wed, 30 Apr 2025 03:05:54 +0000 https://status.buttondown.com/#691e5085d0052ad45b1a4dd0c7fbb5ae69988c9398f7e407ad4ca9f778f77eee p95 response time went down https://status.buttondown.com/ Wed, 30 Apr 2025 02:40:56 +0000 https://status.buttondown.com/#883c128b20584084d2bb8d3bd3bdb4f8b415c76f998606eb118bc827fe92c62c External event backlog recovered https://status.buttondown.com/ Wed, 30 Apr 2025 02:20:54 +0000 https://status.buttondown.com/#883c128b20584084d2bb8d3bd3bdb4f8b415c76f998606eb118bc827fe92c62c External event backlog went down https://status.buttondown.com/ Tue, 29 Apr 2025 20:40:55 +0000 https://status.buttondown.com/#2976efd8a03b8de1c7ca84ceca0f404ed1911e74e4e2bff99b7959d06f1d81d4 External event backlog recovered https://status.buttondown.com/ Tue, 29 Apr 2025 20:20:56 +0000 https://status.buttondown.com/#2976efd8a03b8de1c7ca84ceca0f404ed1911e74e4e2bff99b7959d06f1d81d4 External event backlog went down https://status.buttondown.com/ Mon, 28 Apr 2025 22:10:51 +0000 https://status.buttondown.com/#0455e70352f5a020e5a27fa5086828f8421996c9e57e0a7eb44297ada69123f8 External event backlog recovered https://status.buttondown.com/ Mon, 28 Apr 2025 20:25:43 +0000 https://status.buttondown.com/#798103c2db25441efa368afc84d763aa9e7fbab7db94f346651656e57a09082a p95 response time recovered https://status.buttondown.com/ Mon, 28 Apr 2025 20:05:46 +0000 https://status.buttondown.com/#798103c2db25441efa368afc84d763aa9e7fbab7db94f346651656e57a09082a p95 response time went down https://status.buttondown.com/ Mon, 28 Apr 2025 18:30:47 +0000 https://status.buttondown.com/#0455e70352f5a020e5a27fa5086828f8421996c9e57e0a7eb44297ada69123f8 External event backlog went down https://status.buttondown.com/ Thu, 24 Apr 2025 18:35:50 +0000 https://status.buttondown.com/#1d4ed3e5b5bd4f73aba64c06167ad4031f34be9688bbfc9bccb3bfeb01f04cef p95 response time recovered https://status.buttondown.com/ Thu, 24 Apr 2025 18:25:37 +0000 https://status.buttondown.com/#1d4ed3e5b5bd4f73aba64c06167ad4031f34be9688bbfc9bccb3bfeb01f04cef p95 response time went down https://status.buttondown.com/ Tue, 22 Apr 2025 19:20:25 +0000 https://status.buttondown.com/#7fdb50815b38b83d52ad43faf48df87e85d27aeaeb01a3a5ed6bae2fc8c5f241 buttondown.com/applied-cartography recovered https://status.buttondown.com/ Tue, 22 Apr 2025 19:17:20 +0000 https://status.buttondown.com/#7fdb50815b38b83d52ad43faf48df87e85d27aeaeb01a3a5ed6bae2fc8c5f241 buttondown.com/applied-cartography went down https://status.buttondown.com/ Mon, 21 Apr 2025 19:25:48 +0000 https://status.buttondown.com/#a83030bee704ebb89e690859a5fc603c10a0ef784a197bd63dc9ed7ea33ddc28 p95 response time recovered https://status.buttondown.com/ Mon, 21 Apr 2025 19:05:57 +0000 https://status.buttondown.com/#a83030bee704ebb89e690859a5fc603c10a0ef784a197bd63dc9ed7ea33ddc28 p95 response time went down https://status.buttondown.com/ Mon, 21 Apr 2025 15:40:50 +0000 https://status.buttondown.com/#eb6f8b7b62f0d21379721b5723853c3775b218d11122e8e0c06eff36cd7ea5fd External event backlog recovered https://status.buttondown.com/ Mon, 21 Apr 2025 15:28:55 +0000 https://status.buttondown.com/#9bfb54c60a670449eade9366df6e4e31483f34fab0209846cb16efe8ad3bfd0f buttondown.com/applied-cartography recovered https://status.buttondown.com/ Mon, 21 Apr 2025 15:25:54 +0000 https://status.buttondown.com/#9bfb54c60a670449eade9366df6e4e31483f34fab0209846cb16efe8ad3bfd0f buttondown.com/applied-cartography went down https://status.buttondown.com/ Mon, 21 Apr 2025 15:20:51 +0000 https://status.buttondown.com/#eb6f8b7b62f0d21379721b5723853c3775b218d11122e8e0c06eff36cd7ea5fd External event backlog went down https://status.buttondown.com/ Mon, 21 Apr 2025 15:19:48 +0000 https://status.buttondown.com/#6e9c8257184301f5072fdb7b1e3f456d56825327af1954d94ba4c3f52f965e50 p95 response time recovered https://status.buttondown.com/ Mon, 21 Apr 2025 15:19:47 +0000 https://status.buttondown.com/#36238222adb5efbe611830e2937c15bbab942997f66824f70c0be247e32dd2ac buttondown.com/applied-cartography recovered https://status.buttondown.com/ Mon, 21 Apr 2025 15:10:53 +0000 https://status.buttondown.com/#a3a77c932482b097ad46b6f80c175c48443034ecb99fb482b3accdba9ef575bd External event backlog recovered https://status.buttondown.com/ Mon, 21 Apr 2025 15:00:55 +0000 https://status.buttondown.com/#a3a77c932482b097ad46b6f80c175c48443034ecb99fb482b3accdba9ef575bd External event backlog went down https://status.buttondown.com/ Mon, 21 Apr 2025 14:59:52 +0000 https://status.buttondown.com/#36238222adb5efbe611830e2937c15bbab942997f66824f70c0be247e32dd2ac buttondown.com/applied-cartography went down https://status.buttondown.com/ Mon, 21 Apr 2025 14:55:55 +0000 https://status.buttondown.com/#c0eb6c1f3edf7472d48ceebfd876db86a95d334e2feeb6096ca66773b7f73eca buttondown.com/applied-cartography recovered https://status.buttondown.com/ Mon, 21 Apr 2025 14:40:46 +0000 https://status.buttondown.com/#6e9c8257184301f5072fdb7b1e3f456d56825327af1954d94ba4c3f52f965e50 p95 response time went down https://status.buttondown.com/ Mon, 21 Apr 2025 14:40:44 +0000 https://status.buttondown.com/#d63ec8c232e64687dc2d3917186b5b9cddb38629901e73103db0aa3bffe06536 External event backlog recovered https://status.buttondown.com/ Mon, 21 Apr 2025 14:35:52 +0000 https://status.buttondown.com/#c0eb6c1f3edf7472d48ceebfd876db86a95d334e2feeb6096ca66773b7f73eca buttondown.com/applied-cartography went down https://status.buttondown.com/ Mon, 21 Apr 2025 14:22:54 +0000 https://status.buttondown.com/#740f6aa62be2de628487697ea08564b1d0b3179cf900546a7752a6a018e9a2c0 buttondown.com/applied-cartography recovered https://status.buttondown.com/ Mon, 21 Apr 2025 14:20:53 +0000 https://status.buttondown.com/#740f6aa62be2de628487697ea08564b1d0b3179cf900546a7752a6a018e9a2c0 buttondown.com/applied-cartography went down https://status.buttondown.com/ Mon, 21 Apr 2025 14:20:47 +0000 https://status.buttondown.com/#d63ec8c232e64687dc2d3917186b5b9cddb38629901e73103db0aa3bffe06536 External event backlog went down https://status.buttondown.com/ Mon, 21 Apr 2025 11:40:44 +0000 https://status.buttondown.com/#05f0ad516d9497af2adf69feb500a0688dd37f74aaa56aa156f554c3198296da External event backlog recovered https://status.buttondown.com/ Mon, 21 Apr 2025 11:10:49 +0000 https://status.buttondown.com/#05f0ad516d9497af2adf69feb500a0688dd37f74aaa56aa156f554c3198296da External event backlog went down https://status.buttondown.com/ Sat, 19 Apr 2025 02:40:37 +0000 https://status.buttondown.com/#9edd0fc81c184a5bfb4f01ab43fce35007a14a11ca4e97fc53b78da0440bfea2 External event backlog recovered https://status.buttondown.com/ Sat, 19 Apr 2025 01:30:39 +0000 https://status.buttondown.com/#9edd0fc81c184a5bfb4f01ab43fce35007a14a11ca4e97fc53b78da0440bfea2 External event backlog went down https://status.buttondown.com/ Fri, 18 Apr 2025 22:20:41 +0000 https://status.buttondown.com/#86cbfe388da2c7c03a54443744a1c06cf21066bf7f7f7707e9c059c2bc83a447 External event backlog recovered https://status.buttondown.com/ Fri, 18 Apr 2025 22:10:44 +0000 https://status.buttondown.com/#86cbfe388da2c7c03a54443744a1c06cf21066bf7f7f7707e9c059c2bc83a447 External event backlog went down https://status.buttondown.com/ Thu, 17 Apr 2025 23:50:40 +0000 https://status.buttondown.com/#1727c29931cdb1895f3cd015e1ed9cd0518f28089e171c3bb16cb8d73352d5ab External event backlog recovered https://status.buttondown.com/ Thu, 17 Apr 2025 23:40:39 +0000 https://status.buttondown.com/#1727c29931cdb1895f3cd015e1ed9cd0518f28089e171c3bb16cb8d73352d5ab External event backlog went down https://status.buttondown.com/ Thu, 17 Apr 2025 10:10:51 +0000 https://status.buttondown.com/#9591005648e6a7dfe3bb03e253efe0213860be606019e57553a61d860ca41557 External event backlog recovered https://status.buttondown.com/ Thu, 17 Apr 2025 10:00:47 +0000 https://status.buttondown.com/#9591005648e6a7dfe3bb03e253efe0213860be606019e57553a61d860ca41557 External event backlog went down https://status.buttondown.com/ Thu, 17 Apr 2025 03:30:44 +0000 https://status.buttondown.com/#2a627496e912ff29ecbb2707cb443cb05b2649394152a01dbc2ff88e2b54e564 External event backlog recovered https://status.buttondown.com/ Thu, 17 Apr 2025 02:50:38 +0000 https://status.buttondown.com/#2a627496e912ff29ecbb2707cb443cb05b2649394152a01dbc2ff88e2b54e564 External event backlog went down https://status.buttondown.com/ Thu, 17 Apr 2025 02:49:57 +0000 https://status.buttondown.com/#cb6e517bf16c260a6f1623a0f9ebe221f0f861e04f940ad55dcd4a06ac08970b External event backlog recovered https://status.buttondown.com/ Thu, 17 Apr 2025 01:10:40 +0000 https://status.buttondown.com/#cb6e517bf16c260a6f1623a0f9ebe221f0f861e04f940ad55dcd4a06ac08970b External event backlog went down https://status.buttondown.com/ Thu, 17 Apr 2025 01:03:49 +0000 https://status.buttondown.com/#eb272948cf40b41aea5bcdab8d886bcbcef7e943eaa0bebd4f63f2fa9ffb5b29 External event backlog recovered https://status.buttondown.com/ Thu, 17 Apr 2025 00:14:54 +0000 https://status.buttondown.com/#eb272948cf40b41aea5bcdab8d886bcbcef7e943eaa0bebd4f63f2fa9ffb5b29 External event backlog went down https://status.buttondown.com/incident/547115 Wed, 16 Apr 2025 21:00:00 -0000 https://status.buttondown.com/incident/547115#e3fb3a06a6859995dcb01aa8c5cd27177493f842cf9175822db6b73d8c5e3acf # TL;DR Bad configuration on one of our self-hosted SMTP servers caused a crash that proved difficult to recover from, leaving lots of emails “stuck” in varying degrees – and their being stuck manifested in a slew of unpleasant ways. We’ve fixed the configuration, are investing (literally, right at this very moment) in better tooling and alerting, and are architecting a way to prevent this from ever happening again. ## The gory details Buttondown uses a number of providers to actually send the emails written by authors to their subscribers. In addition to using explicit vendors, we run and maintain our own fleet of servers dedicated to this purpose. We're going to refer to these servers as postal servers, as that's a reference to the great open source project which we rely on. On Wednesday morning, we received some automated alerts from our checker system indicating that our backlog of emails was higher than it should be. After digging in a little bit, we realized the reason it was so high was because each individual email was taking a huge amount of time to send out for one specific server. After a couple more minutes, this server got to the point where all it was doing was trying for a minute and then timing out. (Software engineers reading this might already be getting some ideas of what had happened.) We logged into that server and quickly discovered that the issue was with the database storing messages that were pending delivery. While our initial instinct was thinking that the problem was the overall volume of messages being sent to this particular server, we discovered that the volume was actually secondary to just the overall connection count. We were trying to connect to this database from too many worker threads and it was not set up to recover gracefully or even notify downstream connections what was happening. Once we discovered this, the first order solution was pretty simple. We cycled the database, scaled down the number of workers, and got the connections into a pretty manageable state. The problem we were now left with was that of recovery. We had around 70,000 messages stuck in purgatory. They were technically pending, but some of them, just due to the database connection, were actually correctly sent. Some of them were marked as sent but not actually sent, and so on. We basically entered a fog of war situation where our sources of truth were no longer valid. Our SOP in these cases is to err on the side of caution. Caution in this case means hazmatting that specific server, spinning down all of the workers, leaving all of those messages as pending, and then traffic over to another server or vendor to make sure we don't exacerbate the problem nor accidentally act upon incorrect information. This is exactly what we did. We shifted over traffic, the queue drained, and we resent any emails that we were very, very confident hadn't been sent. We cleared out the problematic server and resumed traffic. ## How we’re fixing it If you've read along this far, you're probably wondering what we're going to do to make this better. The first step, one that is essentially complete by the time we publish this postmortem, is a classic one. Add much more monitoring and alerting. We were over-reliant on the integration and high-level metrics for these servers, which works well when problems are obvious and well-formed, but doesn't work well when they're a little bit more out of the mainstream. To be specific, we already had alerting on pending or stuck messages at a per-server basis. But in order to actually fire those alerts, you needed to have an active connection to the database, which we couldn't have in this scenario. The second one is a little bit broader, which is that we need to do a much, much better job of proactively pushing information about these kinds of spending patterns to you, the author — one of the worst feelings is sending an email and being confused because it's marked as sent, but you haven't seen it in your inbox. We're going to start erring on the side of oversharing about the state of these things so you can proactively poke around within the dashboard and understand what might be causing delays in us getting your emails to your readers. ## Customer impact Over the course of the afternoon, approximately 13,000 subscribers across 40 authors experienced some combination of the following: - Multiple hour delays before receiving a message - Not receiving an email at all (though we’ve redriven these.) - Multiple sends of the same email ## Zooming out To be blunt, we've had too many incidents lately. We've invested a lot in fixing bugs and stability at an object level over the past six months. But we've done a poor job of investing in stability at an end-to-end infrastructural level. The past few weeks have driven that point home. Our most important job as a tool is to reliably send your writing to your subscribers. We have not sufficiently invested in the very boring but very important kinds of observability that we needed to, and we're shifting a lot of our roadmap over the next six months to make sure that our ability to diagnose and resolve these issues is much, much stronger than it has been. If you're still at the end having read through all of this, I know it's not because of rabid curiosity but likely because of frustration because you've trusted us with a job and we haven't been up to the task. But we take this stuff seriously and we're pouring everything we have into it. https://status.buttondown.com/ Wed, 16 Apr 2025 20:07:12 +0000 https://status.buttondown.com/#1f842e347a2f5bc0d2d8028e603e47dde338713c7413efa9af0fd20c8e74e561 buttondown.com/applied-cartography recovered https://status.buttondown.com/ Wed, 16 Apr 2025 20:01:51 +0000 https://status.buttondown.com/#1f842e347a2f5bc0d2d8028e603e47dde338713c7413efa9af0fd20c8e74e561 buttondown.com/applied-cartography went down https://status.buttondown.com/ Wed, 16 Apr 2025 17:23:41 +0000 https://status.buttondown.com/#a5b358898e517ca618330d47d47513cc6174b8fd3acb9f71f2c4806a7443185c Backlog recovered https://status.buttondown.com/ Wed, 16 Apr 2025 17:23:41 +0000 https://status.buttondown.com/#9cb4dc346c244783743c486c78f876540d625580a2021ec33d05af14fadadb26 External event backlog recovered https://status.buttondown.com/ Wed, 16 Apr 2025 15:10:29 +0000 https://status.buttondown.com/#a5b358898e517ca618330d47d47513cc6174b8fd3acb9f71f2c4806a7443185c Backlog went down https://status.buttondown.com/ Wed, 16 Apr 2025 15:10:29 +0000 https://status.buttondown.com/#9cb4dc346c244783743c486c78f876540d625580a2021ec33d05af14fadadb26 External event backlog went down https://status.buttondown.com/ Wed, 16 Apr 2025 14:50:27 +0000 https://status.buttondown.com/#2e4d614f94a61a58139b74cf8f810bd87804a540ecda5c43649038aaed077d80 Backlog recovered https://status.buttondown.com/ Wed, 16 Apr 2025 13:50:41 +0000 https://status.buttondown.com/#2e4d614f94a61a58139b74cf8f810bd87804a540ecda5c43649038aaed077d80 Backlog went down https://status.buttondown.com/ Wed, 16 Apr 2025 13:40:38 +0000 https://status.buttondown.com/#e5f1a906d9d941b7db9f286b096ad90eccdcd4db82f9fec4eadf73564906b971 Backlog recovered https://status.buttondown.com/ Wed, 16 Apr 2025 13:40:37 +0000 https://status.buttondown.com/#f20d88fb0f92e2acc8cbcead0c0ce6871a045f36ba27c7a2d1998e73f6d90090 External event backlog recovered https://status.buttondown.com/ Wed, 16 Apr 2025 13:30:40 +0000 https://status.buttondown.com/#f20d88fb0f92e2acc8cbcead0c0ce6871a045f36ba27c7a2d1998e73f6d90090 External event backlog went down https://status.buttondown.com/ Wed, 16 Apr 2025 13:00:36 +0000 https://status.buttondown.com/#3ddde1b27a9cca8e5c31791a5cc04dc656a45b1ae5450ec1033c3cc577ec4b2b External event backlog recovered https://status.buttondown.com/ Wed, 16 Apr 2025 12:50:40 +0000 https://status.buttondown.com/#3ddde1b27a9cca8e5c31791a5cc04dc656a45b1ae5450ec1033c3cc577ec4b2b External event backlog went down https://status.buttondown.com/ Wed, 16 Apr 2025 12:30:50 +0000 https://status.buttondown.com/#9235af34835541b03ade92b361cc1aa6f1a6cb2c04f221eabd42ee3228c6e3f8 External event backlog recovered https://status.buttondown.com/ Wed, 16 Apr 2025 12:10:51 +0000 https://status.buttondown.com/#9235af34835541b03ade92b361cc1aa6f1a6cb2c04f221eabd42ee3228c6e3f8 External event backlog went down https://status.buttondown.com/ Wed, 16 Apr 2025 11:30:27 +0000 https://status.buttondown.com/#2506adbaf2267edc064a82e588dec4cfc77b283677dd3138dcd5bb482de861c2 External event backlog recovered https://status.buttondown.com/ Wed, 16 Apr 2025 11:20:53 +0000 https://status.buttondown.com/#2506adbaf2267edc064a82e588dec4cfc77b283677dd3138dcd5bb482de861c2 External event backlog went down https://status.buttondown.com/ Wed, 16 Apr 2025 11:20:20 +0000 https://status.buttondown.com/#e5f1a906d9d941b7db9f286b096ad90eccdcd4db82f9fec4eadf73564906b971 Backlog went down https://status.buttondown.com/ Wed, 16 Apr 2025 11:00:56 +0000 https://status.buttondown.com/#2b51a7325da1b80999bdc219a99aa2bdd43c9fe547ed95676536e829559eb1a4 External event backlog recovered https://status.buttondown.com/ Wed, 16 Apr 2025 11:00:56 +0000 https://status.buttondown.com/#d710f1684be0bf4d62c99dafbe2f7417332f867b04f71bc2b8d4ba1ccba9f0cb Backlog recovered https://status.buttondown.com/ Wed, 16 Apr 2025 10:40:55 +0000 https://status.buttondown.com/#d710f1684be0bf4d62c99dafbe2f7417332f867b04f71bc2b8d4ba1ccba9f0cb Backlog went down https://status.buttondown.com/ Wed, 16 Apr 2025 10:40:55 +0000 https://status.buttondown.com/#2b51a7325da1b80999bdc219a99aa2bdd43c9fe547ed95676536e829559eb1a4 External event backlog went down https://status.buttondown.com/ Wed, 16 Apr 2025 06:30:47 +0000 https://status.buttondown.com/#25910a336109e7beff96667359c2277bb8180ae04c4042c995bbc0aa45bd4189 Backlog recovered https://status.buttondown.com/ Wed, 16 Apr 2025 06:20:48 +0000 https://status.buttondown.com/#f84c0f4783da07a393e59400777c92371d77f704194d3367e02793c44ff323a0 External event backlog recovered https://status.buttondown.com/ Wed, 16 Apr 2025 06:10:50 +0000 https://status.buttondown.com/#f84c0f4783da07a393e59400777c92371d77f704194d3367e02793c44ff323a0 External event backlog went down https://status.buttondown.com/ Wed, 16 Apr 2025 06:00:45 +0000 https://status.buttondown.com/#25910a336109e7beff96667359c2277bb8180ae04c4042c995bbc0aa45bd4189 Backlog went down https://status.buttondown.com/ Wed, 16 Apr 2025 01:30:42 +0000 https://status.buttondown.com/#4e95b603328d633e400614cc2a4d470c4b6fd78bd1f36266561c5418478707e9 External event backlog recovered https://status.buttondown.com/ Wed, 16 Apr 2025 01:20:42 +0000 https://status.buttondown.com/#4e95b603328d633e400614cc2a4d470c4b6fd78bd1f36266561c5418478707e9 External event backlog went down https://status.buttondown.com/ Tue, 15 Apr 2025 19:20:46 +0000 https://status.buttondown.com/#dc622e60c68595eac4813448c98f9cf2595a5fc9c3238989c5be05455f378dc1 p95 response time recovered https://status.buttondown.com/ Tue, 15 Apr 2025 19:05:41 +0000 https://status.buttondown.com/#dc622e60c68595eac4813448c98f9cf2595a5fc9c3238989c5be05455f378dc1 p95 response time went down https://status.buttondown.com/ Tue, 15 Apr 2025 18:10:46 +0000 https://status.buttondown.com/#9ed3ac7b8a5780fd203770ee27b5bdcfcd3aa45c511b4769457d9e3c8faacf4c External event backlog recovered https://status.buttondown.com/ Tue, 15 Apr 2025 18:00:42 +0000 https://status.buttondown.com/#9ed3ac7b8a5780fd203770ee27b5bdcfcd3aa45c511b4769457d9e3c8faacf4c External event backlog went down https://status.buttondown.com/ Tue, 15 Apr 2025 17:30:40 +0000 https://status.buttondown.com/#ecda4c598870d6c90ffa66fe739f688e1779222416c21d701bbd4d1f29a1f95a External event backlog recovered https://status.buttondown.com/ Tue, 15 Apr 2025 17:20:38 +0000 https://status.buttondown.com/#ecda4c598870d6c90ffa66fe739f688e1779222416c21d701bbd4d1f29a1f95a External event backlog went down https://status.buttondown.com/ Tue, 15 Apr 2025 17:10:48 +0000 https://status.buttondown.com/#a8025dd0c3870137971644fa6ddba812d450431d660aed3086e5d6351409e8b8 External event backlog recovered https://status.buttondown.com/ Tue, 15 Apr 2025 16:10:47 +0000 https://status.buttondown.com/#a8025dd0c3870137971644fa6ddba812d450431d660aed3086e5d6351409e8b8 External event backlog went down https://status.buttondown.com/ Tue, 15 Apr 2025 16:00:47 +0000 https://status.buttondown.com/#818bd2f710fee60f06f60e9e5649f8dba1ce6f069ab437bc7cb89070b1716cf6 External event backlog recovered https://status.buttondown.com/ Tue, 15 Apr 2025 15:40:46 +0000 https://status.buttondown.com/#ec98db270740c1533da8ecf46314b0138dc71be382f4839545d8063b6ed06b11 Backlog recovered https://status.buttondown.com/ Tue, 15 Apr 2025 15:30:38 +0000 https://status.buttondown.com/#818bd2f710fee60f06f60e9e5649f8dba1ce6f069ab437bc7cb89070b1716cf6 External event backlog went down https://status.buttondown.com/ Tue, 15 Apr 2025 15:19:09 +0000 https://status.buttondown.com/#ec98db270740c1533da8ecf46314b0138dc71be382f4839545d8063b6ed06b11 Backlog went down https://status.buttondown.com/ Fri, 11 Apr 2025 03:55:46 +0000 https://status.buttondown.com/#18e7d6d5c5997df1a679c58230800d0435ba7b27d0d585c88d418d156f5ccc41 p95 response time recovered https://status.buttondown.com/ Fri, 11 Apr 2025 03:47:44 +0000 https://status.buttondown.com/#147ab27dd0430dacb1801547d1ac18df9bb4677571b9d2d66391b6da6565f460 buttondown.com/applied-cartography recovered https://status.buttondown.com/ Fri, 11 Apr 2025 03:40:45 +0000 https://status.buttondown.com/#18e7d6d5c5997df1a679c58230800d0435ba7b27d0d585c88d418d156f5ccc41 p95 response time went down https://status.buttondown.com/ Fri, 11 Apr 2025 03:39:40 +0000 https://status.buttondown.com/#147ab27dd0430dacb1801547d1ac18df9bb4677571b9d2d66391b6da6565f460 buttondown.com/applied-cartography went down https://status.buttondown.com/ Fri, 11 Apr 2025 02:45:41 +0000 https://status.buttondown.com/#f17e0d3a113a08d23a8ac8855a281e8dcc005e7f3fb8e9d6d27d645f36b6a81b p95 response time recovered https://status.buttondown.com/ Fri, 11 Apr 2025 02:38:40 +0000 https://status.buttondown.com/#3e94290f2d11562b816dea851e4ddaf64bf329e11cd5a8a9e95ea1f3b5d4e4e5 buttondown.com/applied-cartography recovered https://status.buttondown.com/ Fri, 11 Apr 2025 02:06:38 +0000 https://status.buttondown.com/#3e94290f2d11562b816dea851e4ddaf64bf329e11cd5a8a9e95ea1f3b5d4e4e5 buttondown.com/applied-cartography went down https://status.buttondown.com/ Fri, 11 Apr 2025 01:35:38 +0000 https://status.buttondown.com/#f17e0d3a113a08d23a8ac8855a281e8dcc005e7f3fb8e9d6d27d645f36b6a81b p95 response time went down https://status.buttondown.com/ Fri, 11 Apr 2025 01:33:10 +0000 https://status.buttondown.com/#b5f4e0e3256cbc3bb3887cba259818228af2df73d234f351f667be485ffc3c3f p95 response time recovered https://status.buttondown.com/incident/543429 Fri, 11 Apr 2025 01:33:00 -0000 https://status.buttondown.com/incident/543429#528fb50f837c45b8067e1a81db3daf8deb8fc22fbaddf84f6f1be26d2e74da12 From 8:38pm EDT to 8:50pm EDT, we were serving 503s for around 75% of our incoming requests. This was purely due to a high burst of traffic that our scaling mitigated (albeit not quickly enough!) We're going to look into the problematic routes and harden their performance. https://status.buttondown.com/incident/543429 Fri, 11 Apr 2025 01:33:00 -0000 https://status.buttondown.com/incident/543429#528fb50f837c45b8067e1a81db3daf8deb8fc22fbaddf84f6f1be26d2e74da12 From 8:38pm EDT to 8:50pm EDT, we were serving 503s for around 75% of our incoming requests. This was purely due to a high burst of traffic that our scaling mitigated (albeit not quickly enough!) We're going to look into the problematic routes and harden their performance. https://status.buttondown.com/ Fri, 11 Apr 2025 00:53:39 +0000 https://status.buttondown.com/#40204bffdfcfccd5e1641331e65492828832ff92eaef8e6a143dd16232bba085 buttondown.com/applied-cartography recovered https://status.buttondown.com/ Fri, 11 Apr 2025 00:40:48 +0000 https://status.buttondown.com/#b5f4e0e3256cbc3bb3887cba259818228af2df73d234f351f667be485ffc3c3f p95 response time went down https://status.buttondown.com/ Fri, 11 Apr 2025 00:39:38 +0000 https://status.buttondown.com/#40204bffdfcfccd5e1641331e65492828832ff92eaef8e6a143dd16232bba085 buttondown.com/applied-cartography went down https://status.buttondown.com/ Thu, 10 Apr 2025 21:10:43 +0000 https://status.buttondown.com/#9d283f5df76efb93e248e8161c14b084f1293d3774811f75b24ab52b6a281bbc External event backlog recovered https://status.buttondown.com/ Thu, 10 Apr 2025 20:20:41 +0000 https://status.buttondown.com/#9d283f5df76efb93e248e8161c14b084f1293d3774811f75b24ab52b6a281bbc External event backlog went down https://status.buttondown.com/ Thu, 10 Apr 2025 20:10:44 +0000 https://status.buttondown.com/#c779e65b35066c45ffedbc04253a7bd2b46b620c8308ab135cad8464244b7122 External event backlog recovered https://status.buttondown.com/ Thu, 10 Apr 2025 19:50:41 +0000 https://status.buttondown.com/#c779e65b35066c45ffedbc04253a7bd2b46b620c8308ab135cad8464244b7122 External event backlog went down https://status.buttondown.com/ Thu, 10 Apr 2025 14:30:43 +0000 https://status.buttondown.com/#5ccf833a2f7bd52aa5be718824cb0e4ad9e40ac924e2b9322490cb7142f37011 Backlog recovered https://status.buttondown.com/ Thu, 10 Apr 2025 13:30:44 +0000 https://status.buttondown.com/#5ccf833a2f7bd52aa5be718824cb0e4ad9e40ac924e2b9322490cb7142f37011 Backlog went down https://status.buttondown.com/ Thu, 10 Apr 2025 09:00:42 +0000 https://status.buttondown.com/#44f128aaceec3696e97dbf9e9a4062335d659f73490a79d4fa3da8ba67e16e74 External event backlog recovered https://status.buttondown.com/ Thu, 10 Apr 2025 08:50:42 +0000 https://status.buttondown.com/#44f128aaceec3696e97dbf9e9a4062335d659f73490a79d4fa3da8ba67e16e74 External event backlog went down https://status.buttondown.com/ Mon, 07 Apr 2025 19:45:33 +0000 https://status.buttondown.com/#71055773180a84414089972aa9b1dfc86c3a0970cc84062d444581ec2f239c5c p95 response time recovered https://status.buttondown.com/ Mon, 07 Apr 2025 19:25:33 +0000 https://status.buttondown.com/#71055773180a84414089972aa9b1dfc86c3a0970cc84062d444581ec2f239c5c p95 response time went down https://status.buttondown.com/ Mon, 07 Apr 2025 19:20:07 +0000 https://status.buttondown.com/#f8fecab15776614d7f00fcc5a5d3da7d0bd4200518957d54d10a009708359881 p95 response time recovered https://status.buttondown.com/ Mon, 07 Apr 2025 18:55:33 +0000 https://status.buttondown.com/#f8fecab15776614d7f00fcc5a5d3da7d0bd4200518957d54d10a009708359881 p95 response time went down https://status.buttondown.com/ Mon, 07 Apr 2025 16:40:37 +0000 https://status.buttondown.com/#b22f2b4713a955e9fa883ad3368f33aa268580c7bf2210f47ff23e60cafebfca p95 response time recovered https://status.buttondown.com/ Mon, 07 Apr 2025 16:25:36 +0000 https://status.buttondown.com/#b22f2b4713a955e9fa883ad3368f33aa268580c7bf2210f47ff23e60cafebfca p95 response time went down https://status.buttondown.com/ Mon, 07 Apr 2025 05:55:46 +0000 https://status.buttondown.com/#e15072ce9b80bf9faec7879c484810f35b3cedaa5a03787e027930f5afa01bc7 p95 response time recovered https://status.buttondown.com/ Mon, 07 Apr 2025 05:45:40 +0000 https://status.buttondown.com/#e15072ce9b80bf9faec7879c484810f35b3cedaa5a03787e027930f5afa01bc7 p95 response time went down https://status.buttondown.com/ Fri, 04 Apr 2025 18:30:52 +0000 https://status.buttondown.com/#bf6eb92ae6b438eb15611aeba0d836da1c7ced7d8720f7d13af44a928502fff6 Backlog recovered https://status.buttondown.com/ Fri, 04 Apr 2025 18:10:37 +0000 https://status.buttondown.com/#bf6eb92ae6b438eb15611aeba0d836da1c7ced7d8720f7d13af44a928502fff6 Backlog went down https://status.buttondown.com/ Thu, 03 Apr 2025 02:50:31 +0000 https://status.buttondown.com/#c5f21bb0f02c37501a4304dfa65ac32e99343339825bd7db6ea506ca994810dc buttondown.com recovered https://status.buttondown.com/ Thu, 03 Apr 2025 02:47:46 +0000 https://status.buttondown.com/#c5f21bb0f02c37501a4304dfa65ac32e99343339825bd7db6ea506ca994810dc buttondown.com went down https://status.buttondown.com/ Thu, 03 Apr 2025 01:38:50 +0000 https://status.buttondown.com/#076e66ce5ded5b833ebb79bcc95c8cfcd5e4a5a439a9c29250a18ccc72801e9e buttondown.com recovered https://status.buttondown.com/ Thu, 03 Apr 2025 01:36:05 +0000 https://status.buttondown.com/#076e66ce5ded5b833ebb79bcc95c8cfcd5e4a5a439a9c29250a18ccc72801e9e buttondown.com went down https://status.buttondown.com/incident/538844 Wed, 02 Apr 2025 18:20:00 -0000 https://status.buttondown.com/incident/538844#c89c920252787b2fc61b8db6be13e52394b29dd7e23de5e32631afaa50858a7a The change has been reverted, and subscribers have been restored to their correct status. There is no action required from authors. There was no impact to subscriber-facing features. https://status.buttondown.com/incident/538844 Wed, 02 Apr 2025 18:20:00 -0000 https://status.buttondown.com/incident/538844#c89c920252787b2fc61b8db6be13e52394b29dd7e23de5e32631afaa50858a7a The change has been reverted, and subscribers have been restored to their correct status. There is no action required from authors. There was no impact to subscriber-facing features. https://status.buttondown.com/incident/538844 Wed, 02 Apr 2025 17:45:00 -0000 https://status.buttondown.com/incident/538844#08ed25b66dbfa6df50c5c1c54d1e36932e151ef19138b9925705cd66626eae3b The change is still being reverted, but we expect that this will be complete shortly. Please continue to refrain from sending until the fix is complete. https://status.buttondown.com/incident/538844 Wed, 02 Apr 2025 17:45:00 -0000 https://status.buttondown.com/incident/538844#08ed25b66dbfa6df50c5c1c54d1e36932e151ef19138b9925705cd66626eae3b The change is still being reverted, but we expect that this will be complete shortly. Please continue to refrain from sending until the fix is complete. https://status.buttondown.com/incident/538844 Wed, 02 Apr 2025 17:15:00 -0000 https://status.buttondown.com/incident/538844#0256ffd8a94bc10f3d2bcfa6b293498a26e979eabd2d27e800b825f11412b279 The Buttondown dashboard and API are mistakenly marking subscribers as undeliverable. We're in the process of reverting the change. https://status.buttondown.com/incident/538844 Wed, 02 Apr 2025 17:15:00 -0000 https://status.buttondown.com/incident/538844#0256ffd8a94bc10f3d2bcfa6b293498a26e979eabd2d27e800b825f11412b279 The Buttondown dashboard and API are mistakenly marking subscribers as undeliverable. We're in the process of reverting the change. https://status.buttondown.com/incident/536193 Sat, 29 Mar 2025 00:18:00 -0000 https://status.buttondown.com/incident/536193#71531a41abee5355cf7f27b03549fb8ef5e593e946661fd272550ff8bebd40a1 We've processed the backlog. https://status.buttondown.com/incident/536193 Fri, 28 Mar 2025 20:26:00 -0000 https://status.buttondown.com/incident/536193#2cd84cb381e65f6540b2f3eec09c3fc70c9311a4c4f7d670974a31ee1a582ff9 We've got a very large backlog of webhooks/automations that need processing. We're scaling up (a lot!) in order to do so; no actions required on your end. https://status.buttondown.com/incident/533622 Tue, 25 Mar 2025 18:51:00 -0000 https://status.buttondown.com/incident/533622#f0a2e8f3b366b9f490abeb29a18b135ec590afac850eefdd52d4fba558728f51 We've confirmed and resent the majority of our affected emails, and are in touch with authors with whom we haven't automatically resent on their behalf. https://status.buttondown.com/incident/533622 Tue, 25 Mar 2025 02:55:00 -0000 https://status.buttondown.com/incident/533622#c1a9b5d10aa84fd801d1c4fa194eb80916c30ebfbd1f63e3df20fd9dcfc8da5a Postmark has reported that [this incident](https://status.postmarkapp.com/notices/bt3ky3r8zlaapqlo-increased-gmail-spam-reports) is now resolved. Our team is continuing to test and monitor. https://status.buttondown.com/incident/533622 Mon, 24 Mar 2025 23:23:00 -0000 https://status.buttondown.com/incident/533622#668b308b22ba35b08a2942c55ac65d1b11b6c880266c1987d9eca2c43ae9c0c3 Postmark has implemented a fix for [the deliverability issue](https://status.postmarkapp.com/notices/bt3ky3r8zlaapqlo-increased-gmail-spam-reports) that caused their IPs to be flagged by Gmail and other providers. We're actively monitoring and will post an update once this is resolved. If you're sending from a custom domain, continue waiting to send large emails until this is fully resolved. https://status.buttondown.com/incident/533622 Mon, 24 Mar 2025 15:32:00 -0000 https://status.buttondown.com/incident/533622#1a14e0101392e76371081483300cafbf8b1fd6a2fb6807803e6a804e07de07b1 We're tracking [Postmark's incident](https://status.postmarkapp.com/notices/bt3ky3r8zlaapqlo-increased-gmail-spam-reports) that is causing a lot of their IPs to be flagged by Gmail and other providers. If you're sending from a custom domain, this likely impacts you; refrain from sending large emails unless you absolutely need to. https://status.buttondown.com/incident/528333 Fri, 14 Mar 2025 18:31:00 -0000 https://status.buttondown.com/incident/528333#44aadb7b6deca2ce55d74475f840c68d53981a725d51944d2a2f469b8dc5e9cb Here is the not-so-fun thing about running an email service provider: you get malicious actors trying to use your infrastructure for, well, malice — phishing, spoofing, et cetera. We have a lot of defenses in place for this, but we detected someone with a relatively novel approach of trying to pass in problematic URLs which we weren't catching. Our other various systems _did_ catch and apprehend this user before they were able to send any emails, but in our haste to push forward a solution quickly we didn't backtest the performance of this solution — in particular, it added a very serious lag to particularly long emails, causing some of them to time out when you went to go and send them (either via the UI or the API.) Normally, when something like this happens we just roll back to stem the bleeding; because this was also a bit of a security/fraud issue, we opted to roll forward and try and fix it live. We are all set now, but apologies if you had issues sending over the last 24 hours; we take that critical path seriously, and hope you understand (and forgive) the interruption. https://status.buttondown.com/incident/528333 Fri, 14 Mar 2025 18:31:00 -0000 https://status.buttondown.com/incident/528333#44aadb7b6deca2ce55d74475f840c68d53981a725d51944d2a2f469b8dc5e9cb Here is the not-so-fun thing about running an email service provider: you get malicious actors trying to use your infrastructure for, well, malice — phishing, spoofing, et cetera. We have a lot of defenses in place for this, but we detected someone with a relatively novel approach of trying to pass in problematic URLs which we weren't catching. Our other various systems _did_ catch and apprehend this user before they were able to send any emails, but in our haste to push forward a solution quickly we didn't backtest the performance of this solution — in particular, it added a very serious lag to particularly long emails, causing some of them to time out when you went to go and send them (either via the UI or the API.) Normally, when something like this happens we just roll back to stem the bleeding; because this was also a bit of a security/fraud issue, we opted to roll forward and try and fix it live. We are all set now, but apologies if you had issues sending over the last 24 hours; we take that critical path seriously, and hope you understand (and forgive) the interruption. https://status.buttondown.com/incident/504518 Fri, 31 Jan 2025 20:52:00 -0000 https://status.buttondown.com/incident/504518#11206448c278e2dfdd6e83e111e726658aa27676e44f27a81af78c7e566804c1 # Post mortem ## What broke? `workerscheduler`, our process for running asynchronous jobs that are scheduled for some date in the future, was hard down for ~six hours. This meant, amongst other things: 1. Outbound emails were down 2. Cron was down 3. Other stuff, but those two dwarf everything else ## Why did it break? At a very high level, our asynchronous worker schedulers work something like this (none of this is bespoke, it's standard RQ): 1. To enqueue a job, serialize the method name, the arguments you want to pass to that method, and a timestamp — store that in Redis. 2. To find a job that should be worked on, pull all of the potential jobs. Sort those jobs by timestamp and start running the first one that is ready. Now, that _the arguments you want to pass_ thing is a bit of a landmine. Consider the following job that we might enqueue: ```python class Email: id: str subject: str body: str @job('five_minutes') def send_email(email: Email, recipients: list[str]): for recipient in recipients: send_email_to_recipient(email, recipient) ``` Enqueuing this means serializing a Python object that looks like: ``` { "method_name": "path.to.module.send_email", "arguments": [ { "class": "path.to.module.email", "id": "1", "subject": "Hi there!", "body": "How are you doing?" }, [ "penelope@buttondown.com", "telemachus@buttondown.com" ] ] } ``` (This is a simplified example, but it's directionally accurate.) Now, the tricky thing about this is that emails can get... large. Our `Email` object stores over sixty columns (four of which are some variation of "the fully rendered body for an email in different formats"), and some authors write _extremely length emails_. Some emails, in memory, are >5MB! Now, with the above toy example — we're sending a single email to a list of two recipients. In reality, Buttondown _batches_ the total recipient list for an email based on a number of factors: an author with 30,000 subscribers might have their email batched into groups of 100 subscribers each (for a total of 300 batches.) And this is where we get into tricky territory. 300 jobs, all containing a 5MB email... now we've suddenly added 1.5GB of memory to Redis, and also now we force the workerscheduler to deserialize 1.5GB of data just to figure out a single job to run. This is, in fact, exactly what happened: the Heroku dyno running the workerscheduler has a memory cap of 512MB, and once we got into this state it kept on trying to read the entire list of jobs, OOM-ing, restarting, ad infinitum. ## Okay, so don't store the entire thing in memory! Right! This is not rocket science. The correct approach is to do something like this: ```python class Email: id: str subject: str body: str @job('five_minutes') def send_email(email_id: str, recipients: list[str]): email = fetch_email_from_db(email_id) for recipient in recipients: send_email_to_recipient(email, recipient) ``` Which then means, in exchange for a slight performance hit (because now we're hitting the database to hydrate the email), all we have to serialize is this: ``` { "method_name": "path.to.module.send_email", "arguments": [ "1", [ "penelope@buttondown.com", "telemachus@buttondown.com" ] ] } ``` We do this almost everywhere... except one place — rate limiting logic per-domain. We happened to trigger _that one place_, and then we were stuck. ## How did we stop the bleeding? By clearing out all of the problematically large jobs from the safety (and large memort size) of my laptop: ```python import django_rq from rq.job import Job STRING_OF_JOB_TO_REMOVE = "send_email_to" QUEUE_NAME = "five_minutes" queue = django_rq.get_queue(QUEUE_NAME) jids = queue.scheduled_job_registry.get_job_ids() jobs = Job.fetch_many(jids, connection=django_rq.get_connection(QUEUE_NAME)) for job in jobs: print(job.description) if STRING_OF_JOB_TO_REMOVE in job.description: queue.scheduled_job_registry.remove(jid) print("Removing!") ``` Once we did that (and re-ran a bunch of stuff to get things flowing again), we were back to normal (albeit with a big backlog!) ## Why did it take so long to notice/fix? The shortest answer is: almost all of our observability runs on crons. If crons are broken, then we're flying blind. But that leads us to... ## How do we make sure this never happens again? - At an object level, we fixed that one problematic code path. We now load emails by ID there instead of serializing the entire object. - At a meta level, we've added a lot of observability through Better Stack so we're not dependent on our own rails. Most notably, we now get paged if no crons have been executed in five minutes: this would have immediately caught the problem. - At an observability level, I've added some internal tooling around analyzing the backlog. It didn't take us too long to diagnose the issue (~thirty minutes), but that's still not great. https://status.buttondown.com/incident/504518 Thu, 30 Jan 2025 16:21:00 -0000 https://status.buttondown.com/incident/504518#aa34ca823374a3f659f84a3b1fc518acc0f467308a34509469baedcda6009371 We've finished redriving the stuck backlog items! We'll follow up with a postmortem later today. https://status.buttondown.com/incident/504518 Thu, 30 Jan 2025 15:14:00 -0000 https://status.buttondown.com/incident/504518#0ab48241fbb0aee39aac6658bc52e965070d7443c96f8ae68c965bcd6296e8bc We have identified the cause and are working on a fix. In the meantime, background work that was stuck has started to be processed again (for example, emails stuck in "About to Send" are now sending). **Importantly, do not send your email a second time. Emails that were sent previously and are waiting will be sent out eventually.** https://status.buttondown.com/incident/504518 Thu, 30 Jan 2025 09:22:00 -0000 https://status.buttondown.com/incident/504518#94e69cf57038bdf7c85dedc45debf01b6e07748d268783def1fba7e7c11d3415 Degradations to our background processing system are causing email sends, scheduled emails, and other background work to be delayed. https://status.buttondown.com/incident/502703 Mon, 27 Jan 2025 18:52:00 -0000 https://status.buttondown.com/incident/502703#7aee67838f20605854cc51e04d1007d0504aa9a6f5eec4f7f2a40847473cfd1b We have identified the root cause of this issue, and a fix has been deployed. https://status.buttondown.com/incident/502703 Mon, 27 Jan 2025 18:30:00 -0000 https://status.buttondown.com/incident/502703#e5e59d38c0d84a4a75f102281b623695873bfc42df78e6136b09f0591daf3293 We are currently investigating reports that some users are unable to send drafts or newsletters https://status.buttondown.com/incident/500977 Fri, 24 Jan 2025 01:15:00 -0000 https://status.buttondown.com/incident/500977#f070dac04ab80b40069a5ae02b4cfbe3dfb1197c08644d94f96772de829ffaff This issue has been identified, and the fix has been shipped. https://status.buttondown.com/incident/500977 Thu, 23 Jan 2025 22:30:00 -0000 https://status.buttondown.com/incident/500977#87c9274e9548356fc1e92d92f0c65dcf97a805bd4b5c74395605de58adeb73b6 We are currently investigating reports that the author-facing dashboard is failing to load for some customers https://status.buttondown.com/incident/497693 Sat, 18 Jan 2025 02:26:00 -0000 https://status.buttondown.com/incident/497693#471a6fa81847d6fe06be8632c985720da105b768175f840304dcbe71fa3e64d8 As written above, we've since recovered from the incident. https://status.buttondown.com/incident/497693 Sat, 18 Jan 2025 02:21:00 -0000 https://status.buttondown.com/incident/497693#3dfd39ee6bd5379934661b7548dd367d25583889ab8e2e009fba26b7181c0e27 From approximately 12:47pm EST to 2:15pm EST, most API requests coming from the author-facing app failed with a 400 error. This occurred because we rolled out a change to our routing and formatting of those requests; amongst other benign changes, we inadvertently changed the formatting of the HTTP method (GET, POST, etc.) of outgoing requests from uppercase to lowercase (get, post, etc.). This change was tested and passed both our continuous integration and manual testing; however, Heroku (our cloud infrastructure provider) does _not_ support this formatting for HTTP methods, and rejected all inbound requests. Once we discovered the issue, we rolled back the deployment and rolled forward a change to correctly capitalize those methods. https://status.buttondown.com/incident/461653 Thu, 14 Nov 2024 20:15:00 -0000 https://status.buttondown.com/incident/461653#48140a0b276496d4fcc9e25332accdb36dc7192c102279116ffa108375f8ade6 We're tracking reports of broken links for folks who have click tracking enabled. We've temporarily switched off click tracking for affected newsletters and are waiting to hear more from our upstream vendor.