I am neither especially clever nor gifted. I am only very, very curious…
When I was 6 years old, the screw of my house door handle broke and I fixed it with my mom’s hair pin. My dad declared: “You will be an engineer one day!”
Maybe it’s because of what he said. Or maybe it’s because I’m naturally inclined. But ever since those early days I’ve been interested in how tools and technologies make things work. It led me to learn programming, which meant I could do cool things like making a simple robot move. With graduate school, a lot of diverse experiences, and dozens of certifications, my path towards DevOps engineering has been eventful and interesting.
To start …
Speed is important for any organization. Developers need their micro release in near real time and operations guys need their end product to be highly stable and secure. DevOps is a cross-functional mode of working and a combination of both mindsets. Hence, people following DevOps methodology tend to be responsible for speed of deployment and stability of product. Today with the DevOps mindset, concentric teams are formed given the increasing importance of security, big data/databases and machine learning operations, so we get DevSecOps, DataOps, MLOps…
Lots of hats
I came to Palo Alto Networks over 3 years ago because I was interested in security. My time here feels like a blur. Creating a highly secure multi-tenant distributed environment without compute and storage affinity has been my focus. I got to solve some really interesting problems around scale, complexity and security. As I evolved, I got an opportunity to wear many different hats 🧢 and it’s been quite thrilling.
I started development in Spark, then got into big data operations and building Hadoop/Kafka clusters along with developing processes/pipelines to increase the efficiency of code deployment. I built some of the most secure PetaByte scale Hadoop clusters with all possible security pillars (TLS 1.2, Kerberos, Encryption) possible in Hadoop (Cloudera flavor).
Wearing the DataOps hat, I implemented next-generation technologies designed for analytics cluster-based storage to ensure that data processing pipelines are highly available and scalable. We built a data lake for our organization. Initially, we had a business problem to segregate different business workloads competing for compute resources. To solve this, we tried running big data applications as containerized workloads and introduced a compute-only cluster (per business unit) which was using data from the central data lake cluster. But this approach did not work, not because of performance but because of operational stability. With our migration to public cloud (GCP) we thought about getting rid of big data compute and storage affinity. So I transitioned from Hadoop to BigQuery + dataproc/Dataflow where the computation in Spark (ML and Analytics) was moved to dataproc and storage of data was moved into BigQuery+GCS. With this migration we were able to achieve our goal.
Wearing the MLOps/AIOps hat, I worked with our data scientists whose prediction models are highly dependent on data quality. One of the biggest operational challenges was that the quality of our non-prod data was not good. The models could not be trained efficiently and the data scientists have to do multiple CI/CD cycles in order to train models with production data. InfoSec policies do not allow prod data to be used in dev environments for model training, so I anonymized the sensitive fields and copied them into non-prod environments to reduce their development cycle time from 1-2 weeks to 2-3 days, or roughly 75-85% reduction! As you can imagine, the data scientists were happy about this. In addition, I redesigned their models in such a way that can take advantage of parallel/distributed computation with reduced cost on cloud.
Security is shifting left
At Palo Alto Networks, following all security principals always takes the driver’s adjacent seat. We add security features from the inception phase of development to build more secure products and reduce the overall go-to-market time. This approach may not work for all organizations but it obviously makes sense for us.
Here’s what I mean. When we needed to transition to GCP, it was not easy as security takes the highest precedence. Everyone started to freak out as the tools that we were using were all of a sudden not usable as they were not hardened per InfoSec cloud guidelines. Extra security hardening was needed for the public cloud migration. All traffic had to be seen from an architecture angle and basic things like setting DNS resolution played an important role for performance intensive applications.
I built a secure perimeter around SaaS services like Big query, GCS, Dataproc, etc… with strict network policies to avoid any public exposure. I also built application endpoints with API gateways and a secret management system, and in the process, helped secure our public cloud setup. For any tool election, security takes precedence over capability.
Keep learning
If, like me, you like building and managing infrastructure along with handling its operations and services, consider the world of DevOps. But keep this in mind:
Dive in: You should have knowledge of all the domains you are going to deal with. In other words, you should be Jack of all trades.
Learning new tools/technologies should be one of your hobbies: In DevOps, there are umpteen new tools/technologies introduced daily so knowing which tool to use to do your work efficiently is key. It doesn’t mean you have to have a tool for each task, but just make sure to have knowledge of tools/technology out there so that you can leverage them. You have to be really comfortable learning and using new tools all the time.
Know your own strengths and weaknesses: You don’t have to be an expert in everything but understanding when to reach out for help is important. You could face multiple challenges/issues at the same time, so multitasking and time management skills are critical.
As the saying goes: Stay hungry, stay curious…
If you enjoyed this post and would like to explore opportunities at Palo Alto Networks, check out our open roles. You can also read more stories like this or learn about our technologies.
