Search for a command to run...
I'm currently an Engineering Manager at Skutopia looking after our Shipping Platform product vertical.
I'm working to scale our engineering teams, give our current engineers the skills to be our future leaders, as well as promoting a positive culture where we can all learn, develop, and grow.
No comments yet. Be the first to comment.
The series will explore the tooling we use at Skutopia to improve and simplify engineering
At Skutopia, we wanted to get in-front of developer experience as we started to scale. After a vendor selection process we settled on using Swarmia as our platform. Swarmia allows us to have engineering surveys as well as DORA and SPACE metrics all i...
The setup During a "Papercut Friday" at SKUTOPIA, I wanted to optimise the startup time of an event sourced service. The setup was as follows: In-memory projections, built every time a NodeJS container starts. A Postgres event store with ~2.5 milli...
It is not just about writing technical documentation.
Skutopia has moved to a GitHub Codespaces first environment for developing applications. Codespaces allow us to have a common development environment where problems are solved once and applied for everyone. Onboarding is a breeze as we no longer have...
At Skutopia we provision and manage our cloud infrastructure (GCP) using Pulumi. In the past I have provisioned infrastructure with tools such as CloudFormation (both when it had to be json 😭 and in the more modern yaml structure), Terraform, and no...
At Skutopia, we wanted to get in-front of developer experience as we started to scale. After a vendor selection process we settled on using Swarmia as our platform. Swarmia allows us to have engineering surveys as well as DORA and SPACE metrics all i...
At Skutopia we have recently migrated off our legacy enterprise platform for managing alerts, on-call and incidents. We now have a modern platform thanks to incident.io. The benefits we receive are immense.
What are the benefits of having all of our alerts, on-call and incidents in one platform?Quite simply, our engineers only need to look in one place to have a complete understanding of the current state of our platform. Should something go wrong, everything within the platform natively interacts. This makes it easy and straightforward to let engineers do their job.
It doesn't hurt that incident.io is easy on the eyes with very straight forward configurations. It's almost impossible to get it wrong.
All of alerts flow into one tab on the webapp. It's very clear to see when we have active alerts. We have full control over what fields we pass in with alerts thanks to the metadata object that we can map in the alert source. This means at a glance our responders are presented with runbooks, alert priorities, links to dashboards, and the effected part of the business.
Once the alert goes off, it flows straight into an escalation path. We base our on call at the moment at the product vertical level. With the escalation paths we simply do a lookup on the effected product and off the alerts go down the correct path. The current active responder will receive a ping via slackbot as well as via the phone app. The on-call schedules sync into a slack user group so it's as simple as tagging the @on-call slack handle should something need to be made visible.
When an alert is raised, a draft incident is prepared in the background. This allows us to quickly move into a full incident response mode should there be a major issue. Should the business need to raise an incident manually, it's as simple as the user running the slack command /inc . The user is presenting with a short form to fill out to give the responder context.
When the incident is created, we receive a dedicated slack channel prepared by the platform that is a single source of information in-regards to the incident. The troubleshooting that takes place in the channel syncs back to the platform. This allows us to run effective post incident reviews.
Updates are sent to a stakeholder slack channel meaning they don't need to be in the weeds to know what's happening, they can follow the concise updates that the incident lead posts out every ~30minutes.
One of the big benefits of the platform is being able to run effective post incident reviews. These are effectively our incident retrospectives where we agree on:
What went wrong?
How did we troubleshoot it?
What was the root cause?
What was the impact on the business?
Cost and hours of engineering
Cost in terms of Ops
Impact on customers in terms of number of effected orders and complaints
What can we do to prevent this happening again?
On the back of the meeting we have a clear list of post incident review tasks to complete. These are logged in incident.io and synced across to our ticketing platform. A priority is assigned with clear definition on priority:
Urgent: Drop everything this must be fixed now
High: We can finish our current task, but this is the next priority
Low: This is a backlog item, there are reasonable work arounds
Recently we have moved to a modern platform for alerts, on-call and incidents. It gives us a single source of truth that is easy to configure, easy on the eyes, and easy for our engineers to use. It takes us through the full life cycle and keeps us accountable for running reviews and completing tasks.