I am managing a large Hadoop cluster and need to identify the most recently updated files within a specific HDFS directory. Standard ls commands show files alphabetically, which isn't helpful for my current audit. Is there a specific flag or a combination of Hadoop shell commands that will allow me to list all files sorted strictly by their timestamp in descending order?
3 answers
To list files in HDFS sorted by timestamp, you should use the command hdfs dfs -ls -t /path/to/directory. The -t flag instructs the Hadoop file system shell to sort the output by modification time, newest first. If you want to see the oldest files at the bottom of the list, you can add the -r flag for a reverse sort, making the full command hdfs dfs -ls -tr /path/to/directory. This is incredibly useful for debugging data ingestion pipelines where you need to verify if the latest partition has been successfully written to the distributed file system without scrolling through thousands of files.
Does this command also work if I want to see the timestamps in a specific human-readable format, or am I stuck with the default YYYY-MM-DD HH:mm format provided by the Hadoop cluster's locale settings?
Use hadoop fs -ls -t. It is the same as the hdfs command and works across almost all versions of Apache Hadoop to sort by time.
I agree with Donna. I've used this in production environments since Hadoop 2.x. It’s the most reliable way to monitor real-time data landing in your landing zone folders.
Matthew, unfortunately, the hdfs dfs -ls command doesn't have a built-in flag to change the date format like the Linux date command does. However, if you need a specific format for a report, you can pipe the output to awk or sed. For instance, hdfs dfs -ls -t | awk '{print $6, $7, $8}' will help you isolate just the date and time columns, which you can then manipulate using standard bash scripting tools to fit your requirements.