I am trying to scrape a list of product names from a search results page. All the names share the same class attribute, "product-title". I’ve successfully used find_elements to grab the list, but when I try to print it, I get a list of object references instead of the actual text. What is the correct loop structure in Selenium-Python to extract and print the visible text for each element?
3 answers
The reason you are seeing object references is that find_elements (plural) returns a Python list of WebElement objects, not their content. To get the text, you must iterate through this list using a for loop and access the .text property of each individual object. For example: elements = driver.find_elements(By.CLASS_NAME, "product-title"). Then, use for element in elements: print(element.text). This is a fundamental concept in Selenium: the driver returns a handle to the DOM element, and you must then query that handle for specific attributes like text, value, or CSS properties. If some elements are not visible, .text might return an empty string; in such cases, you can use element.get_attribute("textContent") to retrieve the text even from hidden elements.
Does your script handle the case where the page uses lazy loading, or do you find that the list of elements is empty if you don't wait for the page to fully render?
You can also use a list comprehension for a cleaner one-liner: texts = [el.text for el in driver.find_elements(By.CLASS_NAME, "your-class")]. It's very Pythonic and efficient!
I agree with Jessica. List comprehensions are excellent for data extraction because they are faster and make the code much more readable. I often use this approach and then immediately join the list into a string or export it to a CSV file for further analysis in data science projects.
Christopher, that’s a critical point for reliability. If the class names are generated dynamically or the content is loaded via AJAX, a standard find_elements might execute before the items are present. I always recommend using WebDriverWait in combination with visibility_of_all_elements_located. This ensures the script pauses until at least one element with that class is visible, which prevents the loop from executing on an empty list and significantly reduces flakiness in automated testing environments.