I am currently automating a data extraction task on a site where all product cards are contained in div elements with the exact same class name. When I use find_element, it only ever returns the first one it encounters. How can I capture all of them separately so I can loop through them and extract the price and title from each individual card? I'm using Python with the Selenium WebDriver and I'm looking for the most efficient way to handle these web element lists without getting StaleElementReference errors.
3 answers
The core solution to your problem is switching from find_element to find_elements. While the former returns a single WebElement, the latter returns a Python list of all matching elements on the page. Once you have this list, you can use a standard for-loop to interact with each div individually. For example, elements = driver.find_elements(By.CLASS_NAME, "your-class") allows you to access elements[0], elements[1], and so on. To avoid the StaleElementReferenceException you mentioned, ensure that the page doesn't refresh or change its DOM structure while you are mid-loop; if it does, you will need to re-locate the list of elements to refresh their references in memory.
Are you planning to perform an action that triggers a page reload inside your loop, like clicking each div, or are you just grabbing the text data from the static HTML?
You can also use CSS Selectors for this, like driver.find_elements(By.CSS_SELECTOR, "div.your-class-name"). It is often faster than XPath for simple class-based lookups.
I agree with Amanda. CSS Selectors are generally more readable and perform better across different browsers in Selenium. It's my go-to for scraping product grids like the one Sandra described.
That’s a crucial distinction, Steven. If Sandra is clicking items that open new pages or trigger AJAX calls, the original list of elements will become "stale." In those cases, I usually recommend finding the total count of elements first, then using a range-based loop to re-find the specific index (By.XPATH, f"(//div[@class='name'])[{i}]") during every single iteration. It’s a bit slower because of the repeated DOM lookups, but it is the most robust way to prevent the script from crashing when the UI is highly dynamic.