By default, recentf-mode cleans its list when the mode gets activated. And that may often be during #Emacs init, potentially slowing things down a little bit.
This behavior can be configured with the recentf-auto-cleanup variable. By setting it to an integer value, it will perform the cleanup by the configured amount of idle seconds (it can also be set to ‘never’ or a fixed time).