Systems Thinking and Where to Find It

TL;DR:

The secret sauce of Principal Engineers is Systems Thinking.
It's a skill that you can’t learn at school.
It's built from four ingredients: The Grind (hard work), The Arena (right job), The Compass (curiosity), and The Accelerant (mentors).
To grow, balance the short-term win of closing tickets and the long-term goal of deeply understanding how everything connects.

In From Feeling an Impostor to Principal Engineer, I shared the story of how an overwhelming feeling of inadequacy forced me to find a path that wasn't about coding faster, but about learning to play a new game.

I remember my first few years as a software engineer. I was good at my job. I wrote clean code, shipped features, closed tickets on time, and even received a “Rookie of the Year” award. But I’d look at the Principal Engineers around me, and it felt like they were operating on another dimension. They had an uncanny ability to connect the dots across the organization, seeing through layers of technology from a single line of code to a global business process in a way I could only compare with magic.

It took time for me to understand, but I eventually realized what I was seeing wasn't some innate superpower. It was a skill that can be learned. This skill is known as Systems Thinking. It is, in my opinion, the most valuable and scarcest technical skill a software engineer can possess. It is the true differentiator between a good engineer and an indispensable one.

After spending more than two decades between academia and industry, I’ve developed a strong sense of why Systems Thinking can’t be taught at school, how software engineers learn it, and why many never do.

What Your Degree Couldn't Cover

Colleges provide a fantastic foundation, teaching the well-known facts and techniques that build upon our common body of knowledge. They introduce aspiring engineers to standards and best practices, and for those who go into graduate studies, they are a place to push the frontiers of scientific knowledge. But they can't teach Systems Thinking.

This skill is fundamentally contextual. It requires two ingredients that a classroom simply cannot provide: years of hands-on experience and a real-life, business-valuable context. I’m sure there are excellent theories about Systems Thinking, but knowing the mechanics of playing a guitar is a world away from playing it well.

Some professions, like medicine, have solved this by creating a long, standardized journey through mastery. Surgeons spend years in pre-med, medical school, and residency before they are considered experts. That model will likely never exist in Computer Science. The reason is simple: the human body hasn't changed much in millennia, but computer systems change daily.

So, if there's no formal curriculum, how do you learn it?

The Recipe for Seeing the Whole System

Let's get one thing straight: you don't have to be a genius. Acquiring this skill isn't about brilliance; it's about a deliberate, strategic approach. It comes down to four key ingredients.

Ingredient 1: The Grind (Serious Hard Work)

There are no shortcuts. Acquiring the depth of knowledge required for true Systems Thinking demands putting in the proverbial hard work. It doesn’t happen overnight, and it doesn’t happen just from 9 to 5.

This is the part nobody likes to hear, but it’s the truth. You have to be willing to put in a lot of extra hours for a long time. It is a common trait in System Thinkers to spend long hours understanding how things work, well beyond what is needed to complete the task at hand. No one will pay you to learn, therefore these hours have the sneaky tendency to pile up on top of the regular working hours. System Thinkers see that deep, often unpaid, effort as the tuition for their own mastery.

Ingredient 2: The Arena (The Right Context)

The second ingredient is being in the right environment. Simply put, you cannot learn how to design and maintain large-scale systems if you are not exposed to them. If you’re working at a company where the biggest challenge is adding a new field to a CRUD form, your growth will have a very low ceiling.

If your current company doesn't provide the necessary context—the scale, the complexity, the high stakes—your only option is to change jobs. My own journey to Principal didn't begin until I joined a Fortune 500 company. The environment was terrifying, but it was the arena I needed to grow.

If you feel you don't have opportunities to learn SystemsThinking at your current role, objectively assess your current environment and decide whether it is time to leave the team or the company.

Ingredient 3: The Compass (Relentless Curiosity)

Hard work in the right context is essential, but it’s not enough. Curiosity is the compass that steers your hard work toward valuable knowledge.

You can work very hard for years developing the same type of RESTful APIs. You might even become very fast and efficient, earning a couple of promotions along the way. But if you never ask the deeper questions, your ceiling will remain relatively low.

A curious mind sees every challenge as an inspiration to research and learn. When you're assigned a ticket, don't just think about the code. Ask yourself:

How does the request travel from the client all the way to my service?
What security layers are in place to protect this endpoint?
What other services are involved in this flow and what roles do they play?
What are the different layers between the hardware and the software I am modifying?
How will this scale to support a 10x increase in load?
How are we observing its performance and health in production?
What's actually happening "under the hood" of the framework I'm using?

To be clear, the purpose is not to find detailed answers to all these questions in your first ticket, but to use every ticket as an opportunity to get a new piece of the puzzle.

Over time, by balancing working effectively on the task at hand and collecting more and more pieces, both your overall understanding of the large system and your ability to solve increasingly complex problems will grow.

Ingredient 4: The Accelerant (Mentors)

This is the final ingredient. You can do it all on your own, but it will be a long and painful road. Mentors are the accelerant in your journey to mastery.

Great mentors do more than just give you advice.

They save you time by pointing you in the right direction.
They open doors by introducing you to the right people.
They enrich your context by pulling you into the right projects.
They unblock you when you hit a wall you can’t see a way around.

But let's be realistic: finding good mentors is hard. The people you want to learn from are incredibly busy, and there aren’t many around. In addition, the mentor-mentee relationship is a peculiar one that requires chemistry and mutual benefit. It doesn't happen by accident.

In a world where these organic mentorships are scarce, a coach can serve as a powerful catalyst, the jump start that you need. Think of a coach not as a substitute for a mentor, but as a dedicated strategist to help you build the skills, mindset, and visibility needed to attract those mentors. But be careful, coaches are expensive, at least the good ones, and there aren’t many qualified coaches out there with real industry experience. If you decide to hire a coach, make sure to find one that you would want as your mentor.

Another approach I have found incredibly effective is what I call “two-way-mentoring”. As I became more knowledgeable of my company's systems, new, talented and experienced engineers and engineering managers joined the group. Consistently spending time helping them navigate the system and the organization helped me learn many valuable lessons from their own personal journeys. Furthermore, my own learning curve steepened dramatically by being around high-performers, as they organically pulled me into the orbit of critical problems and decision-makers.

Playing the Long Game

In conclusion, Systems Thinking isn't a trait you're born with or a subject you learn at school; it's a strategy you execute. It’s a conscious choice to play a different game—one that values understanding over speed.

Treat every task not as a ticket to be closed, but as a thread to be pulled. Use your curiosity to transform routine work into a map of the larger system. Each hour you invest in understanding the "why" behind the "what" is a deposit into a compounding account of knowledge that will make you invaluable.

Seek out the complex arenas and surround yourself with people who challenge you. True growth doesn't happen in your comfort zone; your environment is the ultimate accelerant

It’s a fundamental shift in perspective. Stop competing in the game of who can code the fastest. Start playing the long game of who understands the most.