Gotchas of Streaming Pipelines: Profiling & Performance Improvements
<p>Every streaming pipeline is unique. When reviewing a pipeline’s performance, we ask the following questions: “Is there a bottleneck?”, “Is the pipeline performing optimally?”, “Will it continue to scale with increased load?”<br />
Regularly asking these questions are vital to avoid scrambling to fix performance issues at the last minute. By doing so, one can tune pipelines to perform optimally, consistently satisfy <a href="https://en.wikipedia.org/wiki/Service-level_agreement" rel="noopener ugc nofollow" target="_blank">SLAs</a>, and reduce resource waste.</p>
<p>This article will cover the following topics:</p>
<ul>
<li>Performance improvement process</li>
<li>Strategies to profile streaming pipelines</li>
<li>Common performance problems</li>
<li>General guidelines to improve performance</li>
</ul>
<h1>Performance Improvement Process</h1>
<p>The performance improvement of any software system is not an independent and isolated task but an iterative process. It entails the following steps:</p>
<p><img alt="" src="https://miro.medium.com/v2/resize:fit:600/1*jKxeVtDbGuD3hukViu7-hA.png" style="height:538px; width:600px" /></p>
<ol>
<li>Measure / profile performance</li>
<li>Identify root cause</li>
<li>Fix</li>
<li>Go to step one</li>
</ol>
<p>The process is repeated until you get the desired performance (at the targeted scale) or you exhaust all performance measures.</p>
<h1>Profiling Your Pipeline</h1>
<p>Identifying a performance issue without any profiling tool is a shot in the dark. Profiling is the first step of the process, and requires the right tools. Tools help you identify the most impactful performance issues faster. They can also be integrated with development environments to provide a comprehensive report early in the development lifecycle.</p>
<p><a href="https://eng.lyft.com/gotchas-of-streaming-pipelines-profiling-performance-improvements-301439f46412">Click Here</a></p>