🎯 Goal:
To help you understand the difference between convex and non-convex functions — and why they matter when training machine learning models.
🧠 1️⃣ What Is Optimisation in ML?
At its core, machine learning is about optimising.
You have a loss function (how wrong your model is), and you want to find the best parameters (like weights) that minimise it.
Optimisation = finding the lowest point in a landscape of errors.
This landscape is shaped by the loss function.
📈 2️⃣ What’s a Convex Function?
A convex function looks like a nice, smooth bowl.
It has only one global minimum — no traps, no surprises.
✅ Why it’s awesome:
- Any optimisation algorithm (like gradient descent) will reliably find the bottom.
- You can be confident you’ve found the best possible solution.
Example in ML:
Linear regression has a convex loss surface — which is why it’s easy to optimise.
🌀 3️⃣ What’s a Non-Convex Function?
A non-convex function looks like a mountain range — full of peaks, valleys, ridges, and flat areas.
😅 Why it’s tricky:
- Multiple local minima (not the best solution, but “good enough”)
- Gradient descent might get stuck in the wrong valley
- Optimisers need extra tricks (momentum, Adam, etc.) to navigate
Example in ML:
Deep neural networks have highly non-convex loss surfaces — which is why training them is hard and requires powerful strategies.
⚖️ 4️⃣ Why This Matters in ML
| Concept | Convex | Non-Convex |
|---|---|---|
| Shape | Simple bowl | Bumpy, with multiple valleys |
| Minima | Only one (global) | Many (local and global) |
| Optimisation Difficulty | Easy (guaranteed solution) | Hard (can get stuck or lost) |
| Example | Linear/logistic regression | Deep neural networks, CNNs, LSTMs |
🔁 5️⃣ So What Do We Do with Non-Convexity?
We use:
- Good initialisation (e.g., Xavier, He)
- Smart optimisers (Adam, RMSProp)
- Stochasticity (like in mini-batch SGD)
- Regularisation to smooth the loss surface
These help us find “good enough” minima that work well in practice — even if they’re not perfect.
✨ Final Thought:
Convex optimisation is like hiking in a bowl.
Non-convex optimisation is like hiking blindfolded through the Alps. 🏔️
Understanding this helps you appreciate why some models train easily, while others require careful tuning, patience, and the right tools.
Hi, I’m Anuj — a developer passionate about simplifying technical concepts and building useful tools. This site exists thanks to those students who challenged me, questioned everything, and helped me see things from their perspective. You’re the real MVPs.❤️ Most of the articles here are shaped by my experience as a developer, the books I’ve read, things I’ve discovered online, and — most importantly — the questions my students have asked me.👩💻 And hey, shoutout to Elizveta for passing some of these resources along to your crew — appreciate you!🙌
Learner Reviews
Clear lessons, easy to follow even for a beginner. Very knowledgeable and gives great encouragement to beginner coders. Really helpful and responded to questions quickly. No problem too big or too small. Friendly teaching style meant I didn’t feel embarrassed to ask silly questions.
Louise
I like your jokes. It was always nice how you made us giggle sometimes. Your responsiveness was super appreciated and i loved the additional resources. I think it was nice that you spoke slowly and clearly
Elizaveta
First, Anuj makes efforts to simplify explanations of a concept and gives practical examples in a structured way. Second, he doesn’t just teach us things. He takes time to listen to our concerns and motivates us by sharing his personal experience and thoughts as an experienced software engineer.
Amanda Kartikasari
The presentations you prepare yourself are much more information dense, available and I found some topics are much easier to grasp using them, so grateful for the extra materials you always provide.
Asya Seagrave
Very thorough in terms of showing how to get the code to initially work and what the different errors could be, as well as how to fix them. Very efficient teaching style!
Cerys Cullen
Really liked your teaching style, always very clear and you explained things nice and simply which was great when learning a brand new topic! Loved your videos which were concise, easy to follow and super helpful for going back over to practice with. Loved all the extra resources you provided, your blog, and just in general how helpful and passionate you are about what you do! Thank you so much!
Megan Bradley
Anuj was a clear and concise instructor, who was always happy to help no matter the timing of it. I also liked that he made everyone feel welcome and fostered an inclusive culture, which made our team not afraid to present even if we were slightly unsure if our project work would live up the the standard of the rest of the group.
Deni Hancox
The style of the learning structure has been very constructive and the availabilityof the instructor helps to build confidence and security to try, fail and learn. 5 stars to the instructor style!
Maryl Duprat
Anuj was a fantastic instructor, super informative and knowledgeable. He was very approachable with questions, and clear and concise in his explanations. 🙂
Alice Pinch
Anuj is a great instructor who is very knowledgeable and passionate about what he teaches. He worked well with his co-instructor, which led to a positive learning experience. The course itself is an introductory course but still has a lot of material to cover, but Anuj was very meticulous in explaining concepts to ensure lessons were executed and understood. He is a great listener and always willing to answer questions no matter how obvious (and sometimes silly). I have already signed up for another course and am hoping he will be part of the team!
Pam
Well-structured lessons with the use of diagrams/images to help students understand concepts. Demonstrated examples step-by-step and answered any queries students had.
Maria
Anuj is a great instructor, who was incredibly dedicated to the course, both during teaching hours and after them. I really enjoyed the fun material he posted after the sessions, such as games and fun facts on our Slack channel. I highly appreciate that he was always responsive and patient with any queries or questions that I had, and so any confusions were very temporary.
Raimunda Bukartaite
Really engaging and helpful, always happy to answer any questions I had!
Muskaan
I really liked that the classes involved solving problems during the class and it wasn’t a person speaking alone all the time. I really appreciated the support and commitment of the instructors during the whole course.
Victoria Caballero
Instructions were really clear, and Anuj was very encouraging. He was always open to answering questions and very patient with the class.
Rashida Adekunle
Clear instructions and useful coding habits highlighted.
Asnath Lubanza
Always very helpful – going an extra mile to aid students. Always welcomes questions and will not make you feel bad or stupid for asking.
Maria
Patient when teaching, explains everything clearly and simply for non-techies(!), willing to go slower if us students find it fast or hard to keep up or understand, willing to go over topics and ensure everyone understands. Anuj goes above and beyond for the students despite understanding and learning differences. Outside of class and after class – a passionate and great teacher.
Ameera
Really enjoyed coding alongside you, clear and concise instructions and even some bits that were necessarily on the slides, but were SUPER helpful to us and our understanding of the language
Francessca
Very informative, supportive and friendly. The latter made you very approachable so I felt more at ease asking for help with things.
Amy Mulligan
I found the use cases made things very easy to follow and understand – when I was learning by myself, I was confused at some of the concepts because I couldn’t see how they would be applied, whereas you gave real-life examples that really helped it slot together!
Anastasija Medvedeva
Very supportive, very clear and gives great demonstrations and examples. The extra slides and YouTube videos were also a huge help for me.
Daisy Dobson
I liked that you walked through what you were writing for the code and showed different expamples for the same concept
Fabiha Chowdhury
Used relevant examples of code in the real world, and took the time to explain difficult ideas.
Chloe
I like you are very helpful and teaching the things again and again and make people comfortable to ask you a questions.
Pinar Seker
I think you taught everything really well, thank you! All questions you answered concisely and shared lots of resources.
Ria Kakkad
Your words were organize, not chaotic. You know how to pass the knowledge to students. Thank you
JL
Very clear speaker. Passionate about what you are speaking about. Add your own knowledge into the classes and don’t just read from the script/slides. Explain things in an easy to understand manner. Always available to help.
Caroline Bell
Great teacher, extremely helpful and taught the course in a way which was understandable
Emma McKinney
I liked it, it was very helpful with the tips and notes, also the knowledge was definitely evident so was trustworthy and made me feel comfortable when I thought I asked a stupid question and always had help available.
Sermin Efendi
Really clear, concise and supportive. Broke down complicated problems easily with other examples. The extra content you created with your slides were incredibly helpful.
Nalani St Louis
lots of personality and availability to answer questions/doubts
Liyaan Khoso
Everything Anuj! As a teacher myself, I loved learning from you. You were so patient with us and answered every question we asked so well. You also had a way of making difficult things seem easy. Your passion and dedication showed.
Onyeoma Adigwe
very structured and easy to follow. Very supportive with all the students and always giving insightful advices.
Miriem Shaimi
I really like how you try to simplify and give real-life analogies to coding concepts. Really grateful for all the support you give to us students, especially by addressing our questions and for being available when we are lost. Really grateful for it as it shows how much you care for your students.