The Preliminary Report about our Security Incident on October 21st is now live
There are a variety of things in here that we've done, so it is worth reading thoroughly
A full report is to follow
There are a variety of things in here that we've done, so it is worth reading thoroughly
A full report is to follow
The preliminary report related to the October 21st Security Incident is now available.
blog.railway.com/p/preliminary-…
20
7
124
15,915
Some notes:
- We believe the attacker was using abliterated models with large compute
- We believe they were targeting AI API keys for distillation
- We notified providers, up to "Every key we had for the provider", shortly after unauthorized decrypts occurred, before our initial public disclosure
- We confirmed that the attacker was removed
- We have kept details high level for the security of our users
- We will be posting subsequent follow ups, with more details, as our investigation concludes
- We believe the attacker was using abliterated models with large compute
- We believe they were targeting AI API keys for distillation
- We notified providers, up to "Every key we had for the provider", shortly after unauthorized decrypts occurred, before our initial public disclosure
- We confirmed that the attacker was removed
- We have kept details high level for the security of our users
- We will be posting subsequent follow ups, with more details, as our investigation concludes
2
0
32
2,423
First round of known affected customer disclosures went out last night
Post mortem work continues and should be out at some point today, likely closer to mid-day PST
Post mortem work continues and should be out at some point today, likely closer to mid-day PST
All deploys have now been restored
Affected customers will finishing receiving emails in the next hour or so
Finally, out of an abundance of caution, we will be posting the post mortem around 9am PST tomorrow
9
0
72
6,942
The preliminary report related to the October 21st Security Incident is now available.
blog.railway.com/p/preliminary-…
0
0
13
1,656
All deploys have now been restored
Affected customers will finishing receiving emails in the next hour or so
Finally, out of an abundance of caution, we will be posting the post mortem around 9am PST tomorrow
Affected customers will finishing receiving emails in the next hour or so
Finally, out of an abundance of caution, we will be posting the post mortem around 9am PST tomorrow
Hobby builds have been re-enabled as of 30 minutes ago
Signups, free builds, and trial builds remain disabled for now. We hope to re-enable the latter two later this evening.
Post mortem remains on track for this evening
43
8
393
30,431
We swiftly and broadly provided keys to providers to be revoked the moment we realized *any* keys were compromised
If you received a reset request, and not an email from us, it is likely to be the above
We will cover this in the post mortem
x.com/JustJake/statu…
If you received a reset request, and not an email from us, it is likely to be the above
We will cover this in the post mortem
x.com/JustJake/statu…
@RafUchiha This is expected. We notified broadly for revocation with the labs, providers, etc
We will cover this in the post mortem
1
0
8
1,062
Hobby builds have been re-enabled as of 30 minutes ago
Signups, free builds, and trial builds remain disabled for now. We hope to re-enable the latter two later this evening.
Post mortem remains on track for this evening
Signups, free builds, and trial builds remain disabled for now. We hope to re-enable the latter two later this evening.
Post mortem remains on track for this evening
Builds have been humming for the last 12 hours on Pro and Enterprise
Due to our stance on broad vs narrow notification, we're also scaling to meet need, and aiming to restore all other tiers over the next few hours into the evening
First pass post mortem also later this evening
12
4
151
27,601
Builds have been humming for the last 12 hours on Pro and Enterprise
Due to our stance on broad vs narrow notification, we're also scaling to meet need, and aiming to restore all other tiers over the next few hours into the evening
First pass post mortem also later this evening
Due to our stance on broad vs narrow notification, we're also scaling to meet need, and aiming to restore all other tiers over the next few hours into the evening
First pass post mortem also later this evening
We've begun the process of ramping builds up
We'll continue this into the evening, overnight, and into the morning. Tomorrow we will include a full post mortem
We once again want to apologize for any inconvenience
38
6
239
25,857
As others have pointed out, "humming" isn't exactly correct
Early periods were definitely more stop and start. The last few hours have been more smooth
Hobby now opening back up
General Updates: status.railway.com
Builder Specific: status.railway.com/incident/QH8L8…
Early periods were definitely more stop and start. The last few hours have been more smooth
Hobby now opening back up
General Updates: status.railway.com
Builder Specific: status.railway.com/incident/QH8L8…
0
0
9
2,024
While we believe our definition of impact is quite narrow, we once again aired on the side of notifying quickly and recommending rotation
We understand people are, for a lack of a better word, pissed/disappointed/etc
Working 24/7 in shifts through this. Appreciate the patience
We understand people are, for a lack of a better word, pissed/disappointed/etc
Working 24/7 in shifts through this. Appreciate the patience
8
0
59
4,336
We've begun the process of ramping builds up
We'll continue this into the evening, overnight, and into the morning. Tomorrow we will include a full post mortem
We once again want to apologize for any inconvenience
We'll continue this into the evening, overnight, and into the morning. Tomorrow we will include a full post mortem
We once again want to apologize for any inconvenience
Builds are temporarily paused
We are currently investigating this issue
Status: Investigating status.railway.com/incident/QH8L8…
73
6
372
56,188
Psychotically good marketing
Ramp remains in a league of its own
Aspirational. We need more media/culture/resonance.
Ramp remains in a league of its own
Aspirational. We need more media/culture/resonance.
if you spent any part of your childhood playing tony hawk’s pro skater or watching viva la bam on mtv, you’ll understand why i can’t stop watching this.
one of my favorite nights ever. the best things we make are the ones people feel.
5
4
228
13,225