Stop Training Models in DEV. Train Them in LAB.

Congratulations, ML professionals of all kinds! Only 10 years after Andrew Ng taught everyone about gradient descent, our respectable profession has almost entirely moved away from using the <code>PROD</code> environment for everything. Pat yourself on the back. Now shame on you! Because I glanced over your shoulder just now and saw you training a model in <code>DEV</code>. There’s a better way, people, and it’s called <code>LAB</code>. So here’s what I want to tell you about: <ul> <li>Environments — what they are.</li> <li>How Machine Learning has two very distinct concepts of a development environment, <code>DEV</code> and <code>LAB</code>, that should not be confused (or share the same name).</li> <li>How to organize your team’s work between <code>DEV</code> and <code>LAB</code>, and the implications for other environments.</li> </ul> <h1>Environments</h1> <h2>What are Environments?</h2> Let’s talk for a minute about Environments. First off, let me provide a dense definition of Environment and then unpack it a bit. For the purpose of this blog, an Environment is a runtime that is configured to use a specific group of infrastructure resources for a particular use case. (This definition mashes a few different concepts together, namely runtime, resources, and use case. <a href="https://www.tinystacks.com/blog-post/a-guide-to-stacks-and-stages-on-aws/" rel="noopener ugc nofollow" target="_blank">Here</a> and <a href="https://www.tinystacks.com/blog-post/what-are-stacks-stages-and-environments/" rel="noopener ugc nofollow" target="_blank">here</a> are some useful blog posts separating those concepts for you.) For example, when you’re working in <code>DEV</code> and you start running your code, you’ve probably got your source control checked out to some sort of dev branch. And you’ve got a config file somewhere that points your code to the dev endpoints of any services you use. These things are core elements of your runtime. <a href="https://johndanielraines.medium.com/stop-training-models-in-dev-train-them-in-lab-ad266c6ff3a6">Click Here</a>