I am trying to scrape data from a dynamic webpage where the price information is nested inside a <span> tag. I can locate the element using its class name, but when I try to retrieve the value, it returns an empty string or null. Should I be using .text, .get_attribute("innerText"), or .get_attribute("textContent") to ensure I capture the visible text, especially if the span is updated via JavaScript after the page loads?
3 answers
The most common reason for getting an empty string with .text is that Selenium's default text property only returns "rendered" (visible) text. If the span is hidden by CSS or hasn't fully rendered, .text will fail. To bypass this, you should use element.get_attribute("textContent") or element.get_attribute("innerText"). textContent is particularly powerful because it retrieves the text from the DOM even if the element is hidden via display: none. If you are using Java, the method is element.getAttribute("innerHTML"). Always ensure you have implemented an Explicit Wait using ExpectedConditions.visibilityOfElementLocated to ensure the dynamic content has actually populated the span before you attempt to grab the value.
Are you dealing with a situation where there are multiple span elements with the same class name, and your locator might be snapping to the first, empty hidden span instead of the one containing the actual data?
I usually just use a quick XPath like //span[contains(@class, 'price')] and then use .text. If that returns nothing, I immediately switch to getAttribute("textContent") which always works for me.
I agree with Linda. Using textContent is the most robust way to handle modern frontend frameworks like React or Angular where the text might technically be in the DOM but not yet "visible" to the Selenium driver.
That is a great point, Daniel. I am indeed seeing multiple spans. If I use find_elements to get a list, is there a way to filter that list in Selenium to only return the span that is actually displayed to the user, or do I have to iterate through the entire collection and check the .is_displayed() property for every single element until I find a match?