The ML Playbook Series: A Conversation with Enoch Kan

Welcome to the second edition of The ML Playbook blog series, where we delve into the insights and experiences of industry experts shaping the landscape of Machine Learning Operations.

Our second guest, Enoch, is a seasoned Machine Learning Lead, who focuses on MLOps with some platform engineering too. He is the MLflow ambassador and writes a machine learning blog.

Enoch has lived in three different countries, and his career has spun from software engineering to machine learning, to MLOps and platform engineering. I had the privilege of having a chat with him around all things MLOps as well as what it’s like to migrate to a new country and how to overcome homesickness.

How did you get into MLOps from a more traditional ML background?

It really has to do with my last role because I worked at a relatively small startup, so I was their first full-time, permanent employee. I was a Senior Machine Learning Engineer and then I had to become the Head of Machine Learning. As you can imagine, like in many small startups, you need to do a little bit of everything. So I did the ML research, I wrote code too and then had to figure out how to deploy it. So it was very natural for me to branch out into MLOps.

I think a lot of people think machine learning is just running models to get some kind of results. But it's end to end and a lot of times, it has been in my past, maybe 2-3 jobs. The issue really lies in the data instead of the models. So that's why I transitioned into my role, I needed to work with the data platform team as well to fix some of these data issues.

What advice would you give someone looking to enter into MLOps? 

So what worked for me, might not work for some other people, but with my personality what really benefited my career is that I network a lot. I just like to talk to people and understand what they're doing. And sometimes that leads to career opportunities, sometimes it doesn't, but you never know. It just really enhances your view of the broader market and how the industry is going.

The other thing is to show passion, and show you're very interested in the field of machine learning. You want to enter it right? So have some personal projects. I think a lot of people think that you need to have had your first job in order to get a relatively good position, but sometimes it's not necessary if you show enough passion. When I see CVs that show a lot of passion and they have the necessary skills, I still offer them a fair chance to interview. To me, years of experience is not a good measure of how good of an engineer a person is. I think ultimately if you can show that you're very motivated, then I think some of the hiring managers will still appreciate that.

Out of interest, how do you assess passion on a candidate's CV?

I mean again, this is quite biased because it’s how I scan CVs and maybe other hiring managers will not agree with me, but what I look for is how they structure personal projects. I would actually go into, you know, their GitHub profile, and have a look at their coding style. You can very quickly tell if someone has put time and effort into maintaining a good GitHub profile and some people might have even done some research in a very similar area to what they've applied for job wise, which would really help.  Just seeing how they explain what they've done on the CV, that's very interesting as well.

And then some other people will take on additional roles, like for example, myself. I'm an MLflow ambassador, I'm advocating for open source software so you can really see that on my CV. But that being said, it's not like you just show passion in different areas, you still need to have the relevant experience which comes with, you know, maybe taking an undergraduate course and doing projects and whatnot.

What challenges have you faced in your career, and how have you overcome them? 

The challenges are continuous. The first challenge I ever had in my career was the way I presented myself while interacting with others. Given my cultural background, because I'm Chinese I think for me it's very hard to speak over my superior or feel like I can contribute in a meaningful way before they're complete with their sentences and meetings. What I found very helpful was I had two really good mentors back at my old workplace who were both South Asian and they gave me a lot of leadership opportunities and they put me on the Diversity Board as well. They helped me build my confidence first within my own community. And then they gave me very small group meetings where I became comfortable talking with my peers. And then eventually I think it just builds with time. But you know, from time to time I still have to overcome that, especially as English is my second language. So sometimes my brain doesn't think the same way as a native speaker would.

And then the second thing is related to MLOps and if we’re using the right tooling or not. You can't just give an answer you have got to actually have something to back it up. What's the value you're getting out of it? What's the return on investment? That’s probably a question that I didn't ask as much early on in my career, but now, as I move up in my career, I get asked this a lot and it's not an easy question to answer because you know you can see like many tools like A,B.C,D and they all kind of do the same thing. So sometimes you just have to pick one and go for it. Again, whatever you pick has to have good documentation, right, good knowledge transfer sessions. What are the resources? How long does it take to implement?

How do you choose the best tooling for MLOps then? 

I do research. Which sometimes can be as simple. It's just like a spreadsheet, and you have a few criteria and then you decide. The key here is not to take too long to decide, cause I've made that mistake a couple of times. And if you let it drag on for too long, then you know the damage will be deeper, right? So you need to you know if you think a tool has good value then bring it in as soon as possible. You can even start partnerships or do trials with them, those are free, but again don't end up doing too many trials because that goes the other way around as well if you know you have a limited team with limited bandwidth and you want them to be focusing on the right things. So that's how I currently do it, but there are other ways out there.

So there’s not a tooling at the moment within MLOps that everybody should be using? 

Yeah, there’s no one size fits all. That being said, I am the MLflow ambassador and MLflow is widely used by the community. It is a free open-source product and just like any other open source product, it has its caveats. So, sometimes you have to wait for the community to contribute and there's a cycle. If you want a particular feature and it’s not available, then maybe you need to look at other tools. There are quite a few really good MLOps platforms for experiment tracking, like, weights and biases or Neptune. But again, I think there's a large set of overlapping features, so it really comes down to how much your company is willing to invest, how much maintenance is needed and what the specific use case is, are you training large language models or are you just running regression models? Those are two different questions and the ways they would run on your MLOps framework are very different so need to be considered.

Any ideas on what the future might hold for ML and MLOps? 

I mean if I could predict the future, I’d be a millionaire now. But AI is going to be incorporated heavily in software engineering and will be coming up with decisions or even building out architectures because you know as an engineer myself, I now cope with the help of copilot, and it cuts down my time a lot. There’s a website that tells you exactly what AI tool you can use for a particular use case, and there are tons of them. So while I don't think it'll ever, well never say never, I don't think it’ll fully replace jobs in the short term, it'll make the work more efficient. That's a trend that I see.

And so specifically in the ML OPS side, I think it will help us reach decisions much quicker. For example, you know all that research that I was mentioning or doing comparisons. I'm not going to lie, sometimes I do use GenAI tools, I just chuck in a of couple tools and ask it to provide me with a comparison based on these criteria. So it helps you do the research much faster.

What are the best practices for implementing MLOps? 

While I wouldn’t say implementing MLOps is easy, I think a subsidiary of MLOps is Data Ops, and Data Ops is the part where a lot of people get it wrong, like how you store data, how you do data warehousing. If you get that part right, I think implementing the model inference and stuff like that is relatively trivial and there are a lot of good frameworks around that. So a big part of implementing MLOps is if you can get the Data Ops part right. Then you know the rest comes with it.

Given your experience living in multiple countries, how would you advise someone on relocating for a job, especially if they feel homesick initially?

I think it’s important to keep in mind, that the grass isn’t always greener. You have to find a situation that's stable for yourself and even though maybe in that particular country the salary is lower, if there are fair career opportunities, I think that your career will grow and the money just comes after.

So don't chase money, but chase stability and then money will come after basically. That’s how I decided to move from the US to the UK, even though I felt a little homesick. I mean, wherever home is, I was originally from Hong Kong and then lived in the US for many years and then now I'm here. Who knows where I'll go next. I think the reason why I moved to the UK is because my family's here and I want to be more stable, which has proven to be good for me because when I first moved here is when I first started my ML practical role outside of software engineering. I think it really helped me build up a good CV. And London is a really good city to network and to grow, and it's very international as well. So yeah, look for a stable situation that works for you.

And finally, how did the homesickness, if it did at all, affect your work?

Yeah, I think now it's better because I do have my family here and I have quite a lot of friends, but I do remember at first when I first started my full time role in the US, I did have a lot of free time. It was a really good gig, full time, software engineering role. We’d get off work around four or five and then we just go get a beer after work. But I think because I had a lot of free time, I quickly became very homesick cause all my friends are not in that part of the US or not in the US at all. I pretty much just had to keep a positive attitude and look forward to the next time when my family was visiting but I also used that kind of free time to more activities and projects.

My advice would be to remind yourself of the bigger picture, is it that maybe in a couple years, you'll be in a different situation? If homesickness becomes a really big problem, then you need to ask should I be staying here? And that's the question I was asking myself. That led me to being somewhere more stable. And it’s important to remember mental health and physical health always trump what you're doing to make money or your passion. You have got to take care of yourself first. So that's why I said, if you're feeling extremely homesick and that's kind of affecting your work, then maybe you need to re-evaluate. In the US, I was making decent money and the package was good, I loved the company – it's probably one of the best companies I've ever worked for. But in the end I needed to change my situation and think about where I'm going to be living long term.

If you're seeking top talent in the field of MLOps or looking for exciting career opportunities, don't hesitate to get in touch with our experienced AI & ML recruitment team. We connect businesses with skilled professionals and provide a platform for individuals to explore promising MLOps jobs. Whether you're looking to hire talent, browse job listings, or even feature in our blog series, we're here to facilitate your journey in the world of MLOps. Reach out to us today to learn more!