This is a most excellent place for technology news and articles.
I'm a very good engineer, but so much of my time is consumed fighting with Tekton pipelines and migrating testing frameworks and versions I barely have time to write code. But that's because I can figure that stuff out when I have to. All the code is written by the people who can't figure that stuff out.
Why this isn't two separate jobs I can't understand. Let me do some stuff I'm good at rather than constantly fighting with things I'm not?
This hits the nail right on the head. The point of cloud services is to take away all the overheads of building and delivering software solutions that have nothing to do with the actual business problem I'm trying to solve.
If I want to get a new product to market, I want to spend most of my time making my core product better, more marketable, more efficient. I don't want to divert time and resources to just keep the lights on, like having to hire a whole bunch of people whose only jobs is to provision and manage servers and IT infrastructure (or nurse a Kubernetes cluster for that matter). Managing Kubernetes or physical tin servers is not what my business is about. All this tech infrastructure is a means to an end, not the end itself.
That's why cloud services is such a cost efficient proposition for 98% businesses. Hell, if I could run everything using a serverless model (not always possible or cost effective) I'd do it gladly.
This is quite a trite argument from my point of view. Also, this is from the perspective of the business, which I don't particularly care about, and I tend to look from the perspective of the worker.
Additionally, the cloud allows to scale quickly, but the fact that it allows to delegate everything is a myth. It's so much a myth that you see companies running fully on cloud with an army on people in platform teams and additionally you get finops teams, entire teams whose job is optimizing the spend of cloud. Sure, when you start out it's 100% reasonable to use cloud services, but in the medium-long term, it's an incredibly poor investment, because you still need people to administer the cloud plus, you need to pay a huge premium for the services you buy, which your workforce now can't manage or build anymore. This means you still pay people to do work which is not your core business, but now they babysit cloud services instead of the actual infra, and you are paying twice.
Cloud exploded during the times of easy money at no interest, where startups had to build some stuff, IPO and then explode without ever turning a single dollar of profit. It's a model that fits perfect in that context.
At least where I work, our cloud team is ~35 people who manage the whole thing.
The datacenter team? In the hundreds.
Cloud is not the answer to every infra problem, but the flexibility, time to market, and lifecycle burden are easily beneficial weighed against finops. I’m an Azure engineer myself, it’s no comparison the benefits to a managed solution vs rolling your own DC for a lot of regular business workloads and solutions. Beyond that personally I’ve been able to skill up in areas I wouldn’t be able to otherwise if I was stuck troubleshooting bad cables, rebuilding a dead RAID array, or planning VMWare scaling nonsense.
But those are absolutely not the only 2 levels. Server rental can be managed easily by the same infra team who manages the cloud, for a fraction of cost.
I will say more, the same exact team that spends time managing EKS clusters could manage self-managed clusters and have money to spare for additional hires.
I will say more, the same exact team that spends time managing EKS clusters could manage self-managed clusters and have money to spare for additional hires.
Your suggestions is a large expansion of skillset needed for your alternative to the cloud solution. Your own experience in attempting to hire workers should point to the reason thats a bad idea. You're going to need even higher skilled people, and they are going to ask for significantly more money.
I wouldn't say it's a large expansion of skillset, meaning it's not massive. But yes, indeed it is problematic to find people. It is because this is a vicious circle in which companies are digging their own graves by eliminating a market for those people, which in turn means that those who would want to hire some can't find them easily, leading to outsourcing instead. Do this for 15 years across the whole industry and it stops being an option, which is pretty much where we are today. That said, training and upskilling is always a possibility for companies who invest on their own employees and are playing the long game...
I get you that it's easy to over-provision in the cloud, but you can't return an on-prem server. A cloud VM, just shut it down and you're done.
AWS talks about minimizing undifferentiated heavy lifting as a reason to adopt managed services and I find that largely to be true. The majority of companies aren't differentiating their services via some low-level technology advantage that allows them to cost less. It's a different purchasing model, a smoother workflow, or a unique insight into data. The value an organization provides to customers should be the primary focus of the business, the rest is a means to sharpen that focus.
A cloud VM, just shut it down and you're done.
If this flexibility is needed, and it's an "if", a dedicated server does the same. But even a cloudVM is already lower level compared to other services (which are even more abstract) - like EKS, SQS, etc.
The value an organization provides to customers should be the primary focus of the business, the rest is a means to sharpen that focus.
In my experience this often translates in values that flows to AWS, while the company giving value to customers is stuck with millions of cloud bills each month, and a large engineering footprint that eventually needs to cut, leaving fewer and fewer people working on the product.
That said, I acknowledge that cloud has business reasons to exist, I wrote an entire other post about my hate for it, but I still acknowledge that. However there are some myths that finally are getting dispelled (outsource infra and focus on your product).
I'd like to understand how self managing all the lower level components abstracted by the cloud is saving on headcount. Care to math that out for us?
It depends. An EKS cluster can cost easily 20x what an equivalent cluster costs with same resources. The amount of people necessary to manage it is very close compared to a bare cluster, which depending on the scale can save hundreds of thousands or millions per year, therefore allowing extra headcount.
For example, a company I worked for had a team of 6 managing all their kubernetes cluster on rented dediservers. The infra costed around 50k/year. The same clusters on EKS could be managed by 4 people (maybe?), but would have costed easily 5-600k, especially since they were beefy machines, possibly even more. That amount of money would pay for 7-8 additional headcount in local hires.
Considering that in those clusters there were 40-50 postgres clusters, if moving those to RDS they would have probably looked at millions in cloud bills per year, and the effort to run those dB's once the manifests were developed was negligible (same team was managing them). This was a tiny startup, with limited resources for internal tools and automation development.
So it's not like managing everything can save headcount, it's that not outsourcing everything can save so much money that largely compensates for more headcount, plus you are giving money to real people, who spend local and pay taxes.
I'm not in any way, shape, or form an engineer so I don't really understand the exact details of your post.
However, you post reminded me of a really good episode of a podcast called Hidden Brain. In it the host, discusses the topic of knowledge with a cognitive scientist.
At one point, they talk about how sophisticated technology has gotten that people don't know how to solve problems if that technology brakes, especially since technology is getting so good that it makes fewer mistakes. They use an airplane as an example in which an experienced pilot forgot how to get out of a nosedive and crashed the plane. On a smaller scale, the host mentioned that he has a hard time navigating if his phone's GPS doesn't work.
Its a really interesting listen if you have the chance.
Thanks, indeed I think there are many parallels with other areas. I will check it out.
Or maybe it's just a different skill set
Not when the skillset is essentially outsourced and you are left consuming the product of that skillset.
Understanding is nonnegotiable in security, IMHO.
You can't fail to understand how signature attestation works, if you are implementing it, to make one example I made in the post. Otherwise you end up verifying the signature in the CI (like that person claimed it should be done) and waste the whole effort. You can definitely still outsource the whole infra and scripting to Github, but you still need to understand. The problem is that when you can outsource everything, at some point understanding becomes an extra step.
I get what you’re saying, but also see the other side - these services exist and aren’t ever going away, so the level of knowledge you need about these to use them at least competently is significantly reduced.
What their existence does mean is that there are thousands of developers who wouldn’t ever touch or learn any of this stuff previously are now actually learning it and using it. That’s a positive thing. Not everyone needs to be an expert on the inner workings of everything that a service provides unless you’re specifically looking for an expert.
Also…..people lie on CVs and cover letters. If your ad has buzzwords and technology X, Y, and Z, then you should expect people with little to no knowledge of at least one of those things to have all 3 on their resume.
I partially agree, but not only we are looking for experts of that thing, we are also looking for security experts, and security knowledge is very much meta-knowledge. A software developer might not care at all about - say - how the CI/CD works, because all they care is that the thing builds the code. A security expert generally has a broader scope, and their job is not functional, which means their job is exactly understanding the thing to be able to model the risks around it. So they might not care of all the tools used in that CI/CD or the exact details of the steps, but they should understand the execution flow, the way third party dependencies are pulled, verified, consumed, the authorization model etc.
There is no such thing of security professional who doesn't understand - at least from an academic point of view - the overall setup of a thing they worked with.
If I take the image attestation example I made in the post, I consider the "inner workings" to be the cryptographic details, such as ciphers and their working mechanisms, or the exact details of the way that attestation can be verified offline, or what exactly is computed and how. I am OK with someone not knowing this. But not understanding the whole flow? Well, without this what's left? Copying the 3 lines of code that do something from the Github documentation? Any software engineer can very much do that, what is your contribution as a security specialist?
…..people lie on CVs and cover letters. If your ad has buzzwords and technology X, Y, and Z
Totally agree. It is very likely, although the more people I interview, the more I think that they are not lying from their perspective. It's that people can legitimately make a career today by stitching together stuff with scotch tape, spending years by just by doing that and effectively have little to show for those years. But from their perspective, they might be experienced in that stuff, maybe?
interviewing senior security engineers
Or maybe senior security engineers from 10 years ago were somewhat different from (wannabee senior) security engineers today?
Did you ask them to write 0xD6 in decimal? 😃
That's the thing! I think it wouldn't be conceivable that your "principal engineer" (real position for one of the people) doesn't understand the basic theory of the stuff they are implementing. Now it feels you can instead work years and years just shuffling configuration and pressing buttons, leading to "senior" people who didn't gather actual years of experience.
I don't want to pretend I am outside this logic. I am very much part of this problem myself, having started my career 10 years ago. I do despise cloud services though (if anything, they are super boring), so I tend to work with other stuff. But I could 100% just click buttons and parrot standard and keep accruing empty years of experience...
I agree with your lack of affection for cloud services, but I think your view might be a little skewed here. Does a senior mechanic need to understand the physics of piston design to be a great mechanic, or just gather years of experience fixing problems with the whole system that makes up the car?
I'm a Senior Systems engineer. I know very little about kernel programming or OS design, but i know how the packages and applications work together and where problems might arise in how they interact. Software Engineers might not know how or don't want to spend time to set up the infrastructure to host their applications, so they rely on me to do it for them, or outsource my job to someone else's computer.
Does a senior mechanic need to understand the physics of piston design to be a great mechanic
I would argue that if senior mechanic doesn't understand the physics of piston design at least on some degree he's not a great mechanic. Obviously mechanic doesn't need understanding on metallurgy, CAD models and a ton of other deeper level stuff just like an IT engineer doesn't need to know on a deep level how circuit boards are designed or how CPU die manufacturing process works. But both benefit greatly when they understand why something is built the way it is.
I'm also an systems engineer of sorts and have worked with software engineers. And I've had requests like "Can't you just set 'bind-address = 0.0.0.0 on mysql-server and disable firewall" on a directly internet-facing machine and then received complaints when I'm "making things more difficult" from "senior software" -titles. Sure, I can't write the code they're doing, or at least it would take me a crapload of more time to do that but on the other hand there's guys who have so very narrow understanding on anything they work with that it makes me wonder how they can do their work at all in the first place.
Of course no one can master everything in any field but I find it concerning that a lot of guys just press the buttons more or less randomly until their thing works without any clue on what they actually did and how it might affect on different parts of the house of cards they're building.
But you know what the kernel is. You know that syscalls are a thing, you know what role the kernel performs, you know that different filesystems have different properties (and pros and cons), etc..
You don't need to know the details, perhaps, but you can't ignore the fundamental theoretical concepts of kernel and OS. You might not know the whole detail of the boot procedure, but if your machines are stuck on boot, you know at least what to look for.
Here I was talking about equally foundational topics. There is nothing "above" - say - producing attestations and then verifying them. That's literally all there is to it, but if you don't understand the theory behind it, what exactly are you doing? As as I said, I don't care about the details, I didn't expect someone mentioning ciphers or timestamp authorities, transparency logs etc. All I would expect is "we produce a signature with a bunch of metadata and we verify it where we consume the artifact, so that we are sure that the artifact has the properties attested by the signature".
Not knowing this is like someone claiming that they administer Linux machines but can't explain what network interfaces are or how routing is determined. This is not a question of being expert on different layers, this is just being oblivious to those other layers completely.
You want to hire the "guru", not the "principal". You want to actually ask him to write 0xD6 in decimal, and if he dares to answer "Seriously? Come on now, that's boring", then you hire him on the spot.
But you can't hire only gurus. You need normal seniors, too. Build a normal team around one guru. Maybe build one ultra advanced team around 2-3 gurus, if you really need to invent new and hardcore difficult stuff.
