3 Years of Data, Deadlines and Growth

I’ve been reading the blogs for some years now. I came across the annual reviews on the FreeFinCal site; it’s a shame I couldn’t recall the exact year. My guess would be somewhere between 2018 and 2020.

During the pandemic, I started subscribing to the Substacks. I noticed almost everyone who is regularly writing is reviewing the annual performance. And I always wanted to write one for myself. The birthdays, New Year’s Eves, and financial year-end—all of them provided a chance to write. All these days were an opportunity to sit down and look back at what has happened.

The past week marks the third anniversary in the current role. Also, today is a holiday—it marks a Tamil New Year. It’s a tradition to do good things, and I thought, why not review the work anniversary?

🗓 Work Anniversary Review

As this is the first review, I’m considering entire 3 years.

Date: 2025-04-11 Role: Senior Data Engineer Years Completed: 3 Years

🎯 Highlights of the Past 3 Years

The projects I worked were either migration or automation of manual process.

Project 1: Migration of pipelines between Synapse Workspaces

• This project involved low complex work, but there are high profile users which gave a good visibility.
• I felt Synapse notebooks are lacking features that comes with Databricks.
• Talking to customers and understanding the level of access for them to continue with work turns out to be a mojor pain.

Project 2: Automation project involving multiple, diverse source system

• This project involved moderately complex integrations, but the project is more like a test run for future projects.
• The moderately complex integration with various system turned out to be the showstoppper at certain points.

Project 3: Automation project involving PowerBI, in-house calculation for sustainability

• The current system involves some Excel files. The files have some formula to derive KPIs used for measuring sustainability.
• Understanding the formula in Excel & PowerBI’s DAX code and redoing the formula in SQLs

🚀 Skills Leveled Up

AWS → Azure Transition

1. I was working in AWS before gettng into the role. This role enabled me with Azure. If you read my SM feed you can understand I’m not a fan of Microsft. This notion resisted me to learn the Azure for quite along time, before diving deep.
2. I even completed a certification, even though its very much a fundamental level - DP900.
3. Databricks turns out to be common thread across the Amazon’s & Microsoft’s cloud. The SaaS acted a north star amidst the Microsoft’s ecosystem.
4. My fear was to get locked up in an ecosystem. Fortunately, I’m still using open technologies like SQL & Python. I can port into some other ecosystem, if required.
5. I regret I couldn’t use the Shell script in last 3 years.

Mid → Senior Role

1. I felt this transition tougher than the AWS-to-Azure transition.
2. Apart from coding, this role involves problem-solving. The problem could be technical or with people.
   • The problems involving computer is much easier to solve than the ones involving people.

🧠 Lessons Learned

1. Never postpone things. Punctuality is key.
2. Keep your boss updated; never assume you’re incharge unless officially declared.
3. Git is your only friend. Committing into Git should be continuous process; shouldn’t wait for anything.
4. Whether you like it or not politics is always there. Staying away or rolling with it up to you.
5. If you’re organising meeting announce the agenda before hand. No one likes to be ambushed.
6. People are complex, diverse and biased. Be clear with the communication. When in doubt clarify or get clarified. Assumptions hurts.
7. Make use of AI tools, but don’t be dependent on it.

🔭 Looking Ahead

1. Getting Databricks certified has been a dream for well over 3 years now. May be, I should concentrate on it.
2. Getting into AIML space is work in progress. There should be some progress on this aspect too.

🥂 Closing Thoughts

This has been a very good journey. I should be grateful for the team and bosses.