Python “multiprocessing” “Can’t pickle…”
<p>Python gets ‘…Can’t pickle…’ when using ‘multiprocessing’.<br />
It‘s more than ‘pickle’ issue but OS related issue.</p>
<h2>Background</h2>
<ul>
<li>In previous <a href="https://medium.com/devopss-hole/python-lxml-memory-leak-b8d0b1000dc7" rel="noopener">article</a>, I bumped into the ‘lxml’ issue and the solution is to execute the function using ‘lxml’ in a separate process</li>
<li>Implement a decorator to wrap a function to run in a separate process</li>
<li>The ‘run_on_subprocess’ decorator uses ‘multiprocessing’ Python package</li>
<li>Use Macbook for the Python service development</li>
<li>The service is hosted on a Python Docker service in formal environment</li>
</ul>
<p><a href="https://medium.com/devopss-hole/python-multiprocessing-pickle-issue-e2d35ccf96a9"><strong>Website</strong></a></p>