Building-sentry

As we scale Sentry, we need to expand Postgres capacity by splitting up workloads across multiple primaries.

Cross-region replication is a foundational subsystem in multi-region Sentry. This post explores our design process.

Before splitting our application database and infrastructure up, we wanted to have confidence in where the boundaries would be.

How ASP.NET Core application developers can make the transition from JIT (Just-in-Time) to AOT (Ahead-of-Time) compilation, using the Sentry SDK for .NET as a case study.

Almost 2 years ago, Sentry embarked on a project to bring true EU data residency to Sentry's customers. We decided to do it the hard way.

What is a self identifying file, what are debug IDs and why do we want a fundamental change in the web ecosystem for source maps.

Sentry is a very fast-moving company. In just one month we merged 165 pull requests from 19 authors and changed over 800 files, with a total of over 22,000 additions and almost 10,000 deletions. By updating to Lerna 6 with Nx caching, we were able to reduce our CI run times by about 35%.

At its core, Sentry is a tool that alerts you to defects in your production software. But it does more than blast stack traces into your inbox: Sentry provides powerful workflows to help your team determine root cause, triage issues to your team, and keep tabs on ongoing concerns with comments and notifications. At the end of 2019, the Growth team made it our mission to make it easier for our users to invite their teammates to join them on Sentry. Our theory: improving the user experience of inviting users, as well as democratizing the process to include all team members would lead to a significant increase in team-wide adoption. (Narrator: it did.)

Sentry’s growth led to increased write and read load on our databases, and, even after countless rounds of query and index optimizations, we felt that our databases were always a hair’s breadth from the next performance tipping point or query planner meltdown. Increased write load also led to increased storage requirements (if you’re doing more writes, you’re going to need more places to put them), and we were running what felt like an inordinate number of servers with a lot of disks for the data they were responsible for storing. Here’s a look at how we attempted to understand which database system was right for us and how we adapted our approach when we encountered some unexpected challenges.

Other than Python, JavaScript is the oldest platform that Sentry properly supports, which makes sense considering many Python services (including Sentry itself) have a JavaScript front-end. The system that almost everybody uses to debug transpiled code (and the hopefully apparent subject of this blog post) is source maps. Today, we want to focus on some of the their shortcomings and why source maps cause problems for platforms like Sentry.

Over two years ago, Sentry started supporting its first native platform: iOS. Since then, we’ve added support for many other platforms via minidumps and recently introduced our own SDK for native applications to make capturing all that precious information more accessible. Now, the time has come to lift the curtain and show you how we handle native crashes in Sentry. Join us on a multi-year journey from our first baby-steps at native crash analysis to Symbolicator, the reusable open-source service that we’ve built to make native crash reporting easier than ever.

For most of 2018, we worked on an overhaul of our underlying event storage system. We’d like to introduce you to the result of this work — Snuba, the primary storage and query service for event data that powers Sentry in production.