How to Execute a REST API Call on Apache Spark the Right Way

<h1>Introduction</h1> <p><em>This article uses Python for its examples. For those of you looking for a Scala solution, the theory and approach are completely applicable; check out my Github repo for the Scala source code&nbsp;</em><a href="https://github.com/jamesshocking/Spark-REST-API-UDF-Scala" rel="noopener ugc nofollow" target="_blank"><em>https://github.com/jamesshocking/Spark-REST-API-UDF-Scala</em></a><em>.</em></p> <h2>Setting the Scene</h2> <p>Apache Spark is a wonderful invention that can solve a great many problems. Its flexibility and adaptability give great power but also the opportunity for big mistakes. One such mistake is executing code on the driver, which you thought would run in a distributed way on the workers, such as executing Python code outside of the context of a DataFrame.</p> <p><a href="https://medium.com/geekculture/how-to-execute-a-rest-api-call-on-apache-spark-the-right-way-in-python-4367f2740e78"><strong>Click Here</strong></a></p>