I'm using the threading module to run several API calls in parallel. I realized that thread.join() only waits for completion but doesn't actually return the function's result. Should I be using global variables with locks, or is there a built-in way to "yield" a result? I've heard about concurrent.futures and Queue, but I'm not sure which is considered the industry standard for clean, maintainable code.
3 answers
The modern and most "Pythonic" way to handle this is using the concurrent.futures.ThreadPoolExecutor. Instead of manually managing threading.Thread objects, you submit your function to the executor, which returns a Future object. You can then call future.result(), which will block until the thread finishes and return the actual value (or raise an exception if one occurred). This is far superior to manual threading because it provides a clean interface for results and better resource management. For example, with ThreadPoolExecutor() as executor: future = executor.submit(my_func, arg). This keeps your code readable and avoids the common pitfalls of shared state management.
If I have a long-running thread that needs to return multiple updates or "partial" results over time, would concurrent.futures still be the right choice, or should I look into something else?
You can also subclass threading.Thread and override the run method to store the result in an instance variable, then access it after calling join()
I agree with Emily; subclassing is a great way to understand what's happening under the hood. However, as Sarah (the OP) mentioned, for most production work, I'd stick to ThreadPoolExecutor because it handles the boilerplate for you and makes the code significantly easier for other team members to audit and maintain.
Mark, for streaming data or multiple updates, a queue.Queue is much better suited than a Future. You can have the child thread use q.put(data) whenever it has a result, and the main thread can use q.get() to process them as they arrive. This creates a producer-consumer relationship that is thread-safe and prevents the main thread from idling while waiting for the entire process to finish, which is a common pattern in high-performance Software Development.