In a typical case, a function object will have little or (more often) no persistent state. In such a case, passing by value may no require actually passing anything at all -- the "value" that's passed is basically little or nothing more than a placeholder for "this is the object".
Given the small amount of code in many function objects, that leads to a further optimization: it's often fairly easy for the compiler to expand the code for the function object inline, so no parameters get passed, and no function call is involved at all.
A compiler may be able to do the same when you pass a pointer or reference instead, but it's not quite as easy -- a lot more common that you'll end up with an object being created, its address passed, and then the function call operator for that object being invoked via that pointer.
Edit: It's probably also worth mentioning that the same applies to lambdas, since they're really just function objects in disguise. You don't know the name of the class, but they create a class in the immediately surrounding scope that overloads the function call operator, which is what gets invoked when you "call" the lambda. [Thanks @Mark Garcia.]
与恶龙缠斗过久,自身亦成为恶龙;凝视深渊过久,深渊将回以凝视…