I can second fly.io, I moved a bunch of personal (but "production") projects from Heroku to it recently (Node, Rust, Rails) and the experience has been great. I can also second their Heroku migrator, worked pretty flawlessly.
Overall, my pricing has been about the same or 5-10% less per month.
It doesn't have the same "Click this button to add this third-party service to your dynos, with billing included" eco-system that Heroku has (had?) but my Heroku usage was pretty vanilla so it wasn't an issue for me.
That's the one thing I haven't seen anywhere else. I think technically, for "run simple automatically built containers with little dev effort." there are many better solutions than heroku. The marketplace is the only existing differentiator for heroku, and I don't really understand why no-one seems to be trying to recreate it.
I get "marketplaces" are hard and there's a chicken/egg problem, but is anyone working on it? I think fly has the rest of it sorted and is clearly better. Would love to see the additional services as well.
> I get "marketplaces" are hard and there's a chicken/egg problem, but is anyone working on it? I think fly has the rest of it sorted and is clearly better. Would love to see the additional services as well.
I don't think that particular type of marketplace suffers from the chicken/egg problem, as that problem for me indicates when you must acquire two marketplace participants but it's hard to acquire one without the other.
But when it comes to integrations like this, you could easily write your own integrations as long as the platforms you try to integrate have APIs. Or contract freelancers to build those two. You don't need to have the platform owner to write the integration itself.
I guess I'd call it a "soft chicken/egg" problem rather than "hard".
I do a fair amount of pre-seed and angel investing and I've seen a _massive_ increase in the number of very early businesses that have a "product" that they've been able to build with no/low code tools. It gives non-tech founders a set of options they've never had before, in my experience.
That obviously isn't viable for all early businesses and even the ones it is viable for eventually need to hire engineering teams to build their products, but I love how much more accessible these tools have made shipping something basic.
I'm currently contracting for a client that uses a no-code tool to build their product, and this is exactly my opinion. If you're building a software startup, the only scenario where it makes sense to use a no-code tool (e.g., Bubble), is if you don't know to code. And even then, you will need to replace everything with proper code once you start growing.
Of course, this is a bit different for low-code tools that are used internally etc.
Basically this space is deliver multiples on what engineer team would bring at multiple discounts. It's not really a sustainable model because they are constantly viewing it as a cost center even with the productivity no-code tools. So the moment some open source version is released or they see a cheaper solution they will flock to it especially if you allow easy migration. I've had requests from companies who were doing well with the no-code SaaS but felt they were being held hostage/realize they want an internal tool they own because their requireents always devolves past what no-code tool can do.
This might not be an issue for SaaS who already publish their source code but for the vast majority of no-code/low code, it seems to cater to small to medium enterprises who are constantly looking to reduce their cost centers, even after achieving it an optimal setup.
I am a non-tech founder building a app on building, basically very CRM/Data centric for a specific industry vertical I work in professionally.
I am pushing Bubble to its limits where I have access to approx 150m records with all sorts of other database relationships to many millions of records.
To build this app in a custom software solution, I have been quoted $100k to $500k depending on the backend architecture.
Instead, I am spending around $20-30k in development costs to get my app off the ground to be able to pick up the first paying users... at least that's the goal and I'm a few weeks away from launch.
I would also say that my "MVP" is not like Airbnb's MVP... my hope (and it seems) that it may be functional enough to nearly (70%) replace my existing CRM that I use day to day. Of course, I am planning to transition to a custom software solution once I validate, so I don't see bubble as a long term solution.
I've self taught myself a bit of python and tried teaching myself JS as well off and on, so I am able to generally talk through with software developers what I'm trying to achieve and generally understand the technical things that might be discussed. So I'm not entirely a total non-technical noob.
But the issue with Bubble is there is still a high barrier to actually learning how to use the platform for the average non tech person. It's a black box of sorts and they don't have the best education (unlike webflow). I mean come on, you have to pay $800 to take a bubble sponsored course? Lame. Nonetheless, you learn programming concepts by learning how to build on bubble.
I'll end by replying to the top comment about code being perceived as complex.
Spanish isn't complex. Nor is French. Maybe we can consider Japanese to be more complex. But even then, millions of people speak it just fine. Conjugations in Spanish are complex for a 40 year old English speaker new to Spanish but not complex for a 8 year old native speaker.
But after a certain age, life takes hold, you begin working, and you lose the time and opportunity to spend 100's or 1,000's of hours learning another language. Trying to learn how to code is like this. I sometimes need to carve out 3-6 hours of a day to context-switch away from my busy (non-technical) professional & social life to get back into programming mode.
Low code tools abstract away hours of that complexity you would have to learn, which allows you to start building something functional quicker than you otherwise would have. I know software devs look at low code and say "what's the matter with this crap, I can just spin up a X to do Y in 1 week, this is worthless!". But you are the native Spanish speaker in my metaphor, not the folks learning Spanish way past the days they had time to learn Spanish in college, trying to build the next greatest Spanish hit song to tell their story (i.e. software app!)
> But you are the native Spanish speaker in my metaphor, not the folks learning Spanish way past the days they had time to learn Spanish in college, trying to build the next greatest Spanish hit song to tell their story (i.e. software app!)
To probably strain the metaphor, native Spanish speakers see this like you're struggling with Spanish so you give up and instead decide to learn Esperanto because it's easier and a few people have sold it as being a good alternative to communicate across cultures. You run off and spend 4 months on Esperanto and have a song written and a catchy tune that is well received, but when the time comes to publish an album for the Spanish speaking world, maybe you've got some notoriety built but you've still got to start over from scratch and learn Spanish. For most people, they aren't going to make a hit song on their first try, they're going to fail and have to try again, so learning those Spanish fundamentals instead of a shortcut might have been a better use of time.
This is a really interesting point and it resonates with my experience. When I moved to the US from Europe I was suddenly in a world where people had enormous fridges and freezers and could buy a massive amount of food at a time. They would never have fit in dinky little European apartment before!
This is also my experience for folks that live in NY.
Large fridges are available here; I think there's a lot more to it than fridge size. I suspect that big fridges are popular in the US because in much of the US shopping is extremely inconvenient, rather than shopping being inconvenient _because big fridges are popular_.
Shopping is always inconvenient. Even if the shopping is online and delivered, it’s pretty inconvenient to have to find stuff and schedule a time and take the delivered groceries in when they arrive. One of the features some of the newer 15 minute delivery shops have is allowing you to select only items in stock, which saves all of the actual store time, since you don’t have to be glued to your phone for the inevitable questions.
I walk past two supermarkets and a bunch of smaller shops on my way home from work; shopping is fairly convenient, so I shop basically on demand. What _would_ be inconvenient is doing a "weekly shop"; I don't have a car, and even if I did, the nearby supermarkets don't have parking. So, no need for a big fridge.
I live 100 metres from a store that sells pretty much anything from food to alcohol and then some more. Within 10 min walk I've got access to see much stuff that I would die before trying it all. If I jump in a car, I've got access to enough shopping that I could buy anything from a cake to a sofa. it can be convenient but yes, not everyone would have such access to products.
In Helsinki, my grocery store (~Safeway) is in the building across the street, and open 24/7. In BC, the store was a 10-minute car drive away, and open 9am-9pm.
I had a UK mobile number that had seven consecutive digits (e.g. 079* 444 4444). I got it through a friend who worked in provisioning at a new mobile operator that had just been assigned its new number blocks.
The problem was people would constantly try to "steal" the number. Four or five times a year my phone would just stop working because someone had "persuaded" a call-centre worker at my provider to assign the number to a new SIM they could sell on eBay. Apparently people pay a LOT of money for vanity numbers.
No matter what additional "security locks" they put on my account it kept getting hijacked, so eventually I just got a new (much less interesting) number.
At work the telecom guy was proud of the telephone number he gave me ("very easy to remember!" he said). Well, it turns out it was one digit off of the local Alcoholics Anonymous help line. I had digit "1" and AA had "7". I got a lot of drunken phone calls and voice messages. All depressing since they were asking for help. Figured it out after a few calls, but from now on I'll take the ugly telephone numbers.
Many years ago I worked at a pizza place. The town I was in had numbers in two different exchanges (the three digits between the area code and the final four digits in US numbers). Pretend the pizza place had the number 123-4567. The other exchange in that town was 321.
Naturally, the number 321-4567 just got some person's house. That person set up their answering machine message to say that "Due to a rash of prank calls, we are no longer accepting phone orders" without actually specifying who isn't accepting phone orders. Naturally people would show up in person highly confused about this.
In China, that would be the unluckiest phone number ever. My 30 story Beijing apartment building lacked floors 4, 14, and 24 due to superstition about 4 meaning death.
That sounds absurd. I can't believe a civilized society would exclude so many floors! In the US where we are completely reasonable and rational we only exclude the 13th floor. I used to work on the "14th floor" (actually floor 13) and I thought this was hilarious.
It might also be a trick on the building owner to make their building look taller. They also exclude the 13th, so that’s a total of 4 floors excluded, but it is only 30 stories, so they get to exclude 34 as well so the top floor is 35.
I feel like if I was superstitious about floor numbers I'd still not want to live on the 4th/14th/24th floor regardless of what they called it. It's still the 4th/14th/24th floor.
Apparently it's pretty common for tall buildings in NYC to skip a few floors so that the top floors have higher numbers. I was grateful for this when my friend's building's power was out and I had to take the stairs to the 42nd floor.
I had the same problem. Same type of 07 number as you - all consecutive. It was too much hassle. Also, with that number and my 07969696969 number that I thought was hilarious, I just had infinity prank calls all day, and worst of all, all night...
It still requires more effort to move a phone number to a new SIM than to convince an airline to reset your password.
I hate that SMS is so much relied on (in particular as it fails so bad in travel settings), but I think we're stuck with it for a long time before we get alternative that work at least half as much for standard people that don't buy into ecosystems.
it still prevents drive by attacks. without 2fa, having accounts is basically free (take a list of leaked passwords, and try them on 20 sites). with 2fa, it will probably take 15-30 minutes per account of relatively adept social engineering. it won't stop a targeted attack, but it will prevent 95%
The GP is talking about SMS for 2fa. Much better to use an app like Google Authenticator or a secure token which doesn't rely on 3rd parties (mobile phone companies) being secure.
We all agree that 2fa is good, that is a moot point.
oh yeah, authenticator apps and hardware keys are way better than sms, but I worry a little about demonizing sms 2fa too much since plenty of (especially older) people don't have smart phones, and are never going to use a hardware key. since any 2fa is a huge step up, I think sms based 2fa is likely good to promote in tandem with the better methods. (that said, it drives me insane that most banks only offer sms based 2fa. that should be illegal)
Do NOT use Google Authenticator unless every account you use it for has an alternate MFA option (backup codes, etc) that you've confirmed work. It does not sync to your Google account and there is no way to back it up (even manually). The moment your phone gets stolen, breaks or dropped in a river, you will learn a very quick lesson about MFA backups/alternates.
> Four or five times a year my phone would just stop working because someone had "persuaded" a call-centre worker at my provider to assign the number to a new SIM they could sell on eBay
And they were unable to add a note to that account to the effect of "don't do X under any circumstances"?
They put all kinds of locks on the account but they never worked, my assumption was that people who worked at the provider (who could unlock anything) were the ones stealing it!
Assuming that its similar to att, there is an audit trail, but noone will care to look at it for you. They will give your number back and consider it solved.
Besides that a caller might press one too many or less amount of times the same number. It is all about muscle memory, and between address books and clipboards you might only have to enter it once (or not even that!).
That being said I still remember my old phone number I had as kid but I cannot figure out to remember my wife's (acquired number in 2015 aka the age of smartphone was into effect).
I like the blog post and broadly agree with the conclusions, but I want to double down on something in the article:
> Shared pain
I've worked in some orgs with very large monorepos (1000s of developers working in a single repo) and broadly have had positive experiences with them, but this 'shared pain' was by FAR the biggest drawback I experienced. When things went wrong with the monorepos they tended to go VERY wrong and effect EVERYONE. Multiple incidents of the monorepo just crushing productivity for 1000 person engineering orgs, for a period of time.
That's not to say I think monorepos are bad, it's 100% context dependent whether they make sense for your org in my experience, but I learned that the same tradeoffs you get with architectural monoliths/distributed systems often apply to multi/mono repos as well.
Same experience. It's interesting because we've had great success on our internal design lib's monorepo compared to the overall application. Which makes sense in some regards.
There's a split between our orgs. DocuSign runs a mono repo. But the big product I work on has its own giant mono repo. Design lib is another giant mono repo comprising general front end things also.
This is much less of an issue for Sweden and Finland than other non-NATO members, as we're an EU state and article 42.7 of the Lisbon treaty obligates other EU members to assist in case of military attack.
So any Russian military retaliation that occurred between Sweden announcing they intended to join NATO but before they officially did would, in effect, be an attack on the whole EU block. Which is tantamount to a direct attack on NATO.
Finally, Sweden has several bi-lateral defence agreements with countries like the UK and US. It's obviously not technically the same as being a NATO member, but many people in the Swedish defence establishment regard it as closely equivalent.
Assist doesn't do it for me. I find it near impossible to believe that were Finland attacked, the Central European militaries would be doing anything different from what they're doing now. Assist would equal hand them some equipment and root for them.
I think you serialise misinterpret the sentiment in Central Europe. Several people are already seriously considering if more intervention is necessary. However, the fear of nuclear WW3 is keeping that in check. If Putin was to attack attack Finland or Sweden it would be clear to everyone that he is going to continue further and further.
Apart from that, after this war (no matter if he wins or not) the Russian military would be in no state to attack anyone except maybe moldavia. In fact there was a leaked FSB memo just two days ago, which was highly critical of the invasion and estimated that even if they win they would need 500,000 permanently in Ukraine to "keep" it. That is significantly more than half of the active military in russia.
could you link to the FSB memo? I really want to read it.
that jives with my armchair understanding of Russia's forces as well. they simply don't have the resources to take and hold much more than Ukraine. drafting extra people would be very unpopular. their economy can't handle growing the military, either. they just aren't in the state Germany was when they invaded Poland.
maybe they can threaten nukes, but even if NATO doesn't meet that challenge I can't imagine Russia successfully taking the Baltics with the threat of nukes alone.
NATO's most important member is the US, which isn't party to these European agreements. The same escalation risks that preclude the US fighting for Ukraine would probably prevent the US from fighting for Sweden.
Sweden has to join NATO to get the unambiguous commitment of the US, and to put the decision of running escalation risks on the Russian side.
Treaty text specifies: Aid and assist. Likewise, the second paragraph specifies the NATO remain foundation of NATO member states' security.
It remains untested if "aid and assist" means fighter planes fighting sorties, boots on the ground, or nuclear umbrella. There is very little formal structure for EU military operations, because most member states coordinate via NATO structures.
North Atlantic Treaty is very clear: it says that all member states are in war if one is attacked and specifies use of armed force. While it hasn't been tested either, NATO has the genuine military organization, armed forces and nuclear warheads to implement it.
> So any Russian military retaliation that occurred between Sweden announcing they intended to join NATO but before they officially did would, in effect, be an attack on the whole EU block. Which is tantamount to a direct attack on NATO.
People really should read the NATO treaty. It fits on about 2 sheets of A4. Article 6 makes very explicit what constitutes an attack on a NATO member, and it's not "an attack on another EU country that's not in NATO". It's on the territory of /NATO members/.
"article 42.7 of the Lisbon treaty obligates other EU members to assist in case of military attack."
It's a sentence in a document.
The legitimacy of these 'mutual defence treaties' is a function of legitimacy on the ground, history, disproportionate power etc..
Wait to see what happens if Georgia joins the EU, then the leader does something dumb, Putin II invades ... will Europea attack? Can they? What will they do? Can they agree on a response? How to coordinate?
EU is a political body with some other aspirations, but the 'Hard Power' is NATO.
Not sure if it matters at this point if Sweden is a member, because I believe that Russia invading Sweden or Finland would be the same thing as invading NATO.
I led a lot of data and analytics work for pharma companies on clinic trials and the point about complexity is hard to overestimate. A pretty good analogy for bringing a major new drug to market is comparing it to building a new space flight system, individually it might have one less zero on total cost, but that's more of a factor of humans bringing dozens of drugs to market at any one point so there are economies of scale rather than it being any less complex.
Also, the article does a good job of describing one side of clinical trial complexity, finding participants and getting them through the funnel, but there's a whole other source of complexity in clinical trials that mean you need massive global efforts: different global regulators have different criteria for 'approving' clinical trials. An indicative example, the Japanese regulator needs to see evidence that the drug was tested in Japan during a trial. They're one of _many_ national regulators that say that.
So even if you _could_ find all a trials participants in one country you likely wouldn't be able to get it approved by many national regulators.
There are 100% valid epidemiological reasons, but often these differing national rules are there for policy purposes outside of drug safety, for example, if you mandate trials need to take place in your country it provides a nice investment in biotech for your economy or by creating specific rules as a lever to control cost in your national healthcare system (e.g. NICE in the UK).
At least in the US, the article discusses a valid but frustratingly incomplete step per-nation.
There's a broken incentive system for doctors: they are in small/regional academic centers and get rewarded for recruiting, but mostly when there are few other participating doctors (and thus limiting # hospitals & included communities). You don't get the pharma relationship/funding, no academic recognition for being one middle author over many, no time for understanding all the trials, etc. For blockbuster trials like COVID, the care ROI potential is obvious, but few are like that.
Stuff like cancer is a pathologically bad case bc you often do need that wider net, even when going for an ultimately small patient group. You don't know which hospitals people with the markers being targeted will show up at, nor if they'll fill your diversity goals (... which pharma doesn't actually want as part of the negotiated trial setup, another story). But even if a national register identifies a regional patient + their care provider, chances are, that's not enough for the doctor to enroll the patient.
National registers, or at least more smoothly federated ones, are a good step to decreasing some of the patient identification cost. For cancer, with growing regional genome databases, especially so. But there are several big carrots + sticks doctors face when choosing which trials the system wants them to pursue, and the COVID stuff was able to skip much of that.
Source: daily dinner conv with someone doing the treatment/prescribing, trial enrollment, and working on one of the US's biggest national registers, including to use data to solve targeted trials, yet still struggling.
Many drugs work differently for different genetic pre-conditions. Making sure that a drug works for the majority of your population is a valid goal.
The current climate (due to the pandemic) is to throw many regulations over board. However, many of them have been paid for in blood. And finally pharma companies can't be trusted. Not at the slightest.
I've hired quite a few security folks in my time (some with criminal convictions) but my answer is an unhelpful one: it depends.
If you have a criminal conviction it's unlikely you'll get through the screening process with a regulated business (like banking, insurance, pharma etc) due to some 'out of the hiring managers hands' constraints those industries have. I've seen exceptions to this in the past, where a senior manager strongly advocated for the exception, but it's _very_ rare.
I've worked with several security people with criminal convictions in the past at non-regulated, FAANG and FAANG-like tech companies. They also usually have policies in place to prevent hires with criminal convictions, but the exception process there is easier, particularly in security teams where these convictions are more likely to occur in strong candidates.
The biggest concentration of folks with backgrounds like yours have been at security consultancies, in my experience. Combined with the experience you mentioned with bounties, that would be the place I'd spend most time looking. You might still get rejected from some, for example those with customers that require criminal background checks for employees or security clearance you couldn't get, but there are still quite a large percentage where you could find work. Personally, I've had conversations with external consultancies who say things like "I know you require criminal records checks on all our employees, which we're happy to do, but I want you know >50% of my team will fail them".
A couple of other things:
- No matter where you work, with your background there might be some kind of 'restriction' placed on what you work on and/or how you work (e.g. can't work on project Type X or must work from Office Y). If you do get through a process, ask about this before joining, as it might have an impact on how much you'd enjoy the role.
- Be open about your background. You sound like you would do that anyway, but the more open you are the better, you don't want this to be a surprise to people. What you're looking for is a strong advocate on the hiring team, so building trusting relationships with people will be important.
Don't be too down on yourself, you might have made some bad decisions, but you sound like a talented professional. The criminal justice system exists for people to serve their punishment and then move on with their lives. There are companies that will be delighted to hire you because of your skills. Your road may be a little tougher than for others, but that doesn't mean you can't end up professionally happy, fulfilled and well compensated.
Sort of similar to the above, will rule you out of certain roles with certain businesses but if you're open and honest about it you'll be able to find something. The OP having computer hacking (and more importantly, fraud) on their record will make it harder than a DWI, as a DWI isn't something 'work related' (unless you drive for a living, obviously).
The amount of time passed is also something I've seen have an impact (e.g. if you had a DWI 20 years ago vs one 6 weeks ago)
A good point, and may help in this case, but often when onboarding with a payment processor (especially at the 5-10m a month level of the OP) the majority of the time/complexity isn’t the technical integration.
The complexity is usually compliance (eg KYB, sanctions checks, AML) and while payment processors have done a lot to reduce the time and complexity it can still be a long and painful process.
In my experience (I’m CTO at a payment network) technical integration is <10% the time for a merchant doing this kind of volume moving processors.
Finally, what you described has happened. A lot of payment processors have been heavily influenced by Stripe’s DX and align relatively closely, or at least closely enough to make transition easier. The challenge is that few even medium sized merchants just use the core ‘move money’ APIs that are ‘easily’ replaceable, they use things like reporting/reconciliation, anti-fraud and a host of other products that make up an ‘ecosystem’ of products that in combination is very hard to replace quickly.
I know we’ve all collectively accepted that DNSSEC is a terrible, complicated blight on the world but I still find it incredible that that an organisation with Slacks resources and access to expertise can’t make it work.
You say Slack, and I agree, that's telling, but you have to add to that AWS itself, which had a DNSSEC bug in its wildcard record support as well. Slack and AWS together couldn't make this feature work. Further: the open source tooling Slack (like most places) relies on for deployment is also DNSSEC-hostile: one of their problems is that Terraform's Route53 provider doesn't safely disable DNSSEC once enabled. It's a mess everywhere you look.
I think another interesting question here is why Slack bothered in the first place. As was pointed out on the other DNSSEC thread today: practically nobody in the technology industry uses DNSSEC in the first place. Presumably, Slack did DNSSEC (they don't anymore!) in service of FedRAMP compliance. Why? Slack has one of the most popular products in all of computing. What bad thing was going to happen if they said "nah, we're going to go with Cloud.gov's recommendation and not this FedRAMP document"?
> Presumably, Slack did DNSSEC (they don't anymore!) in service of FedRAMP compliance. Why? Slack has one of the most popular products in all of computing. What bad thing was going to happen if they said "nah, we're going to go with Cloud.gov's recommendation and not this FedRAMP document"?
As just one example, it's tremendously difficult, if not impossible, to sell your cloud-based SaaS to Navy customers if you have open FedRAMP compliance issues that you aren't at least working to address.
I say "compliance" instead of "security" for a reason as well, as "compliance" truly runs the show in Navy cybersecurity. And if you want to sell to that market (and it's hardly just Navy who runs this way), it's easier to check the checkboxes than it is to argue about whether NIST is right or cloud.gov is right.
Gotta be Fedramp compliant to do business with the US government. Even worse, you have to be Fedramp compliant to work with anyone who works with the US government. From a business (if not an engineering) standpoint, there's plenty to gain in going through the motions
I don't know about FedRAMP, but with other government requirements, the easiest way to get an exception was to fail badly at implementing the retirement.
When the DOD tried to mandate Ada, lots of projects were bid as Ada, then switched to C++ at the very first sign of any trouble whatsoever. I would 100% believe it if someone told me that this horrible rollout could be leveraged into an exemption from needing DNSSEC
We had to do DNSSEC (for a couple of "system relevant" services) too.
Was it a hard requirement? No, but the fat fingered audit companies really like to tick that "should" box green and would be more lenient with other debatable findings, so it was suddenly "in our best interests" to comply.
It's a business decision. Good luck selling software subscriptions to federal agencies without FedRAMP compliance.
I'm pretty surprised that slack doesn't have a more robust testing network. Is it really that hard to set up another DNS on Route53 for staging these changes? Idk, but that type of thing is the least you can do if you want some FBI agents to discuss active investigations on your chat platform...
None of this, none of it at all, has anything to do with Slack's ability to safely host conversations from the FBI. Whatever challenges they have with that are entirely orthogonal to this stupid performative stunt DNS configuration.
(There's a whole thread here, and more on Twitter, getting into the actual details of what FedRAMP and NIST require here, and engaging with the fact that Slack is the only large tech company in the past several years to have attempted to flip the DNSSEC switch on.)
I work for a company that maintains DNSSEC on our FedRAMP deployment. It's not unreasonable to ask for signed DNS records if feds are going to hit them.
Your blog post makes the supposition that DNSSEC is only being pushed as an alternative means of security to CA for TLS. While it makes a compelling case that this isn't realistic, there are other security concerns that occur from the compromise of DNS records. If the government is going to use a DNS record, it should be signed by a zone owner.
Slack is actually a good use case for this security enforcement, because they maintain a handful of domains that are extremely authoritative for their messaging service[1]. If you can't maintain a security protocol on four domains that are crucial to the operation of your service, you maybe aren't cut out to supply software for the government.
I've done security work for products deployed at DOD and in other sensitive agencies, and had firsthand experience with USG infosec, and the idea that the USG sets any kind of useful standard for infrastructure security is risible.
Unfortunately, the GSA product market is its own bubble, as is people who work in IT for the USG in any capacity, and so it's easy to see how people with limited exposure to modern industry practice --- experiences almost wholly gated through vendors that snake through the GSA acquisition process --- might believe themselves to be operating several levels above where they actually are.
I would take Slack's security practice --- their infrasec, their corpsec, their software security, the whole shebang --- over anything done in any USG agency. Slack is better at this than their USG clients are, full stop. And Slack, while strong, is far from the S tier of industry security teams.
Well, the Slack security team seems to think that DNSSEC is important. Even for their workspace domains.
I just want to hammer home the point that requiring service providers to get their DNS records signed by DNS zone owners is a reasonable ask for USG software service vendors. Even if DNSSEC isn't capable of securing the whole internet.
DNSSEC is utterly unimportant. Practically no major security team on the Internet enables it --- not Amazon's, not Google's, not Facebook's, not Microsoft's, not Apple's, not Oracle's, not IBM's, not Cisco's. The argument that DNSSEC is somehow necessary for secure infrastructure is an extraordinary claim, and it requires extraordinary evidence.
Well, the operational requirements of commercial entities may be different than the federal government. Many of the companies that you mentioned offer FedRAMP services (with maybe the exception of Apple and Facebook), and they probably reckon with the spec on some level, even if they are not employing it internally. It is also pretty clear that Slack is going to implement it soon - they are going through all this trouble to allow to provide their signature every workspace get's a subdomain feature on FedRAMP. They really don't have to do that. Or maybe they do, in which case I would argue that it is probably good practice to be able to interrogate the DNS records they maintain.
Either way, this argument is starting to become political. Is Facebook a role model for cybersecurity, and keeping data out of the wrong hands? Or do NIST researchers know better? Neither - the government outlines its security requirements, and private companies play ball to compete for their business. And if a federal agency wants to be able to prove a DNS record's authenticity, even if it is maintained by a vendor, even if that isn't sufficient to secure their infrastructure, that's their prerogative.
No tech company is infallible. All of them have outages, some lasting hours, even days.
Complex systems can and will fail. Try to do better, of course, but let’s acknowledge that perfection will always exceed our grasp. The world will continue to turn regardless.
One day it might just be your turn to break production.
I agree with your points about DNSSEC (disclaimer: I have not had the pleasure of having to implement it myself in infra), but was attempting to communicate that DNSSEC isn’t the only area of ops that folks get exposed to these sorts of unknowns or edge cases, and that no amount of resourcing enables you to avoid these issues. For Slack, it was DNSSEC. For Roblox, Consul. Facebook/Insta, software defined BGP. Akamai, DNS.
Perhaps I did not read the room appropriately. Mea culpa.
"It turned out that some resolvers become more strict when DNSSEC signing is enabled at the authoritative name servers, even while signing was not enabled at the root name servers (i.e. before DS records were published to COM nameservers). This strict DNS spec enforcement will reject a CNAME record at the apex of a zone (as per RFC-2181), including the APEX of a sub-delegated subdomain"
Slack's second attempt wasn't a DNSSEC problem. Slack depended on a permissive fallback of revolvers when encountering a plain DNS protocol error. It is similar to how some websites in the past relied on permissive browsers implementation when facing broken HTML/JS/CSS. Slack fixed their broken DNS as a result of this.
Slack's third attempt was not the fault of Slack but rather a software bug at Amazon. I would make the argument that Amazon's primary product isn't DNS services, but they did fixed their bug after this.
The general conclusion I get from the article is not that DNSSEC is broken, nor that is too complicated. It is that when doing changes with your core infrastructure to make it more secure, bugs that may have been laying dormant might pop up and bite. I am sure some people has had that experience in domains outside of DNS.
You are not wrong, but by steering clear of DNSSEC, Slack would not have had the outage they did.
What one can't ignore is the underlying chicken-and-egg problem that DNSSEC must overcome: Not many DNSSEC deployments and hence not much of it has been tested in the real-world, which results in colossal outages despite the attention of some of the most qualified engs, including the ones running one of the largest nameserver deployments in the world.
TLS and WebPKI has had a similar, perhaps even more painful route to ubiquity. So, this problem isn't unique to DNSSEC. What isn't working in DNSSEC's favour is, the world has not just moved on, but it has built solutions atop DNS' weaknesses, like it once did with IPv4 and NAT. Internet's strong network-effects coupled with its heterogeneity, make battling "the System" an even harder proposition.
I know HN has collectively accepted but every time I'm associated with an organisation that pays for a penetration test it comes in as a high risk finding, so much so that I've given in to deploying it to avoid sitting with non-technical managers doing the "here's why I disagree" all over again. Outside of this group I definitely feel like I'm on my own in that view.
Overall, my pricing has been about the same or 5-10% less per month.
It doesn't have the same "Click this button to add this third-party service to your dynos, with billing included" eco-system that Heroku has (had?) but my Heroku usage was pretty vanilla so it wasn't an issue for me.