I am working with a dataset containing over a million rows and I need to find the frequency of each unique entry in a specific character column. I have tried using a for-loop but it is incredibly slow. Is there a built-in R function that can handle this quickly, or should I be looking into external packages like dplyr or data.table for better performance on large-scale Data Science projects?
3 answers
For standard tasks, the built-in table() function is the most direct way to get frequencies. Running table(my_vector) returns a named integer vector where the names are the unique values and the values are their counts. However, if you are already using the tidyverse, dplyr::count() is much more flexible because it returns a data frame (tibble) which is easier to pipe into further visualizations. For truly massive datasets where performance is critical, the data.table package is king. You can use setDT(df)[, .N, by = column_name] to get counts almost instantaneously, as it is highly optimized for memory efficiency in advanced Data Science workflows.
How does the table() function handle missing values like NA? Does it exclude them by default, and can we force it to show them in the final count?
If you just want the total number of unique items without the specific counts, use length(unique(my_vector)). It's much faster than generating a full frequency table.
I agree with Nancy. If the goal is just a count of categories, unique() is the way to go. But for Kimberly's original question about frequency, I’ve found that sort(table(x), decreasing = TRUE) is the most helpful snippet for immediately seeing the most common values in any categorical variable.
Steven, by default, table() ignores NA values. To include them, you must add the argument useNA = "always" or useNA = "ifany". This is a vital step in data cleaning; otherwise, you might underestimate the amount of missing data in your sample. In dplyr, the count() function will include NA rows by default as its own category, which I personally find much more transparent for initial data auditing.