Meet Kiran, Data Engineer At Choco

Hi Kiran. Looking back, what was it that made you want to join Choco as a Data Engineer?

I first discovered Choco on LinkedIn and was immediately drawn to the company's vision of moving towards a sustainable future and reducing food waste. It’s a mission very close to my heart and one of the reasons why I applied.

Over the course of the interview process, I had the opportunity to meet different people and it was really those conversations that sealed it for me. I felt they could really support me in my personal growth. That was the second reason why I really wanted to join Choco.

On my resume, you can see that I transitioned from Data Science to Data Engineering. Even though I was a Data Scientist, I naturally gravitated towards work on infrastructure. I wanted to explore the data engineering domain further and in more detail, which is why I chose data engineering.

You’ve been with Choco for over a year now. What surprised you the most about working here?

One thing that has surprised me is how focused and driven everyone is toward the company's mission and goals. Despite being a startup, Choco is less about the "startup-ey" perks like a kicker table and more about working towards a common goal. It's inspiring to work with so many like-minded people who are all on the same wavelength. I think we’ve seen a lot of growth here over the last year. For me personally, as well.

Before we dive deeper into career growth, let’s talk about the Data Platform team. How would you describe it to those who might not know?

The data platform team helps analysts, individuals & teams, and our users make data-driven decisions by providing them with the information they need. We’re responsible for setting up the infrastructure behind providing data to support decision-making. We build and maintain pipelines to get data from different sources, transform and clean it and then make it available to end-users, both internal and external.

But it's not just building that is part of our job. We also talk to stakeholders about their requirements and understanding of the business use cases. We have a good mix of technical problem-solving, with some stakeholder management as well. All of this makes our day-to-day quite interesting. Let’s look at reports for example. They contain aggregated information from different aspects of our product. We need information about orders, users, etc. All of this data needs to be retrieved from our services, transformed, cleaned, and aggregated so that it can be used in the most valuable way.

Interesting! Can you share what aspects of Choco’s data platform are working well?

I think our data platform is solid and has some great features that are working well for us. One thing I appreciate is that it can handle a large amount of data, which is important as our business grows. We use Spark on an EMR cluster for data processing, which helps us make sense of all that data.

Once the data is processed, we use DBT to turn it into something that's useful for the business. All my colleagues have access to this data, which they can use to build reports, and dashboards, or make decisions.

Another thing I appreciate about our platform is that it can work with other systems and tools if we need it to. It's flexible and can adapt to our needs.

I’m also confident we have a solid system in place to keep our data safe. This is really important because we rely on this data to make decisions that drive the business forward.

Overall, our data platform is a well-defined setup that's working really well for us.

Great to hear! What are you currently working on?

The Data Platform team works very autonomously and we drive our own projects. At the moment, however, I’m assisting a colleague with a project that provides our vendors with information on their customers.

Last week, we had what we call Excellence Days. It’s an initiative that was launched to focus on tackling technical debt. We do this once a quarter to improve the health of our data platform.

During the Excellence Days, I pair-programmed with one of my colleagues to upgrade Python, because we all know Python 3.7 is reaching end-of-life in a few months. We had to upgrade about eight or nine repositories while making sure that all of the existing systems and pipelines are not disrupted.

We operate on different tasks like Spark and DBT-based transformations, batch predictions, lambda functions cleaning data in our data streams, etc. It was a bit of a task to make sure that everything is working properly, but we succeeded in the end!

On a typical day, I’ll start by reviewing open pull requests. I then move forward with the tasks that I planned for the day, which is typical work to progress towards accomplishing our initiatives - like the vendor project I mentioned earlier.

How would you generally describe the culture of the Data Platform team?

It’s great! We all get along really well both technically and socially, which makes it easy to collaborate and have fun outside of work. We have different technical backgrounds and experiences, so it's nice learning from everyone on the team.

We have a good balance between having fun and being creative while also taking our work very seriously. We're all focused on accomplishing our goals and delivering results. And we're not afraid to speak up if we see something that needs fixing or if we have a better idea.

On the social side, we try to have team events almost once a month. Recently, we went go-karting, and we have a big team event coming up where we'll be cooking together, which I'm looking forward to.

Not only do we collaborate with each other, but we work very closely with the BI and Analytics teams at Choco. The Data Platform team ensures that the tools we integrate into the stack are user-friendly and easy to use by all the different teams in the company. We constantly get feedback from them and improve our tooling this way. Apart from team events, we also have a monthly get-together to discuss the impressive things that we are all working on.

Go-karting? Sounds amazing! Alright, Kiran, let’s talk professional growth. What’s been the most valuable thing you’ve learned during your time at Choco?

Absolutely. When I joined Choco, my main motivation was to grow technically. I considered myself relatively inexperienced, so I wanted to work with different technologies with technically strong colleagues and accustom myself to them. Over the past year, I've had the opportunity to do exactly that!

I've grown a lot technically from my exposure to modern infrastructure as well as my collaboration with colleagues who have a strong technical mindset. We not only contribute to the existing infrastructure but also try to make it better. This means that I'm always learning something new on a daily basis, and I'm really enjoying it.

What are you currently learning?

Upgrading Python has been a bit of a challenge. One of the biggest problems we faced was that our CI and CD pipelines didn't run as expected. My colleague and I were surprised that we struggled with GitHub Actions, out of everything we upgraded. We brainstormed and asked ChatGPT and other colleagues for help, but we weren't reaching the exact solution we wanted. After some time of hacking, we accomplished the solution, which is always a nice feeling.

Another huge learning for me was to get involved in a project that dealt with sending emails using Python. It was new to me that email templates written in HTML make use of HTML tables. With this, I got accustomed to using the SendGrid API as well as tweaking my HTML and CSS skills.

I am also involved with the development of a solution to improve observability over DBT models in Airflow. This project aims at increasing pipeline reliability and team productivity by enabling easier observability, issue monitoring, and diagnosis. Our goal is to streamline the analytics engineering teams’ workflow, allowing them to focus on more valuable tasks and deliver better results. This project has a lot of touchpoints across different technologies, and I am sharpening my existing skills and working on them.

It sounds like you're learning a lot that's closely tied to your work. Are you learning anything else that’s outside the scope of your role?

One thing that comes to mind is chatGPT. We've had two different Lunch & Learn sessions just talking about chatGPT, how we can use it for our day-to-day work, what we should not use it for, and what we should use it for. These sessions are not just for technical people but for the entire company. We're sharing knowledge and making sure that everyone knows how to use it since it’s become so popular. These Lunch & Learn sessions are, again, company-wide and happen almost every fortnight with topics ranging from diversity to ChatGPT.

Let’s wrap it up with one final question. What do you want candidates to know who apply for a role on the Data Platform team?

To be honest, I would suggest they just apply. I say this because this was my exact experience joining Choco. Our interview process is very open. We have conversations to learn about each other and our technical experiences. Have a look at our vision, mission, and product and if you feel aligned, apply! The interview process will be a learning experience for both the candidate and us.

We're always looking for new talent, so please get in touch with us on LinkedIn or check out our open roles here!