NuGet Package Restore vs Keeping Everything in a VCS

We're having a big argument here over whether to keep all project dependencies in a VCS (Mercurial, in our particular case) or rely on NuGet restoring them.

I'm a strong believer in that everything should be versioned, binary dependencies included -- this is why we have a Version Control System. Others here say that what we have is a Source Control System, and hence binaries don't belong here.

Arguments for keeping them in a VCS and against NuGet Package Restore:

• Once repository is cloned locally, I don't need to be Internet-connected anymore


• I can reliably hg update to any revision and all dependencies are guaranteed to be there


• Nothing fancy is required from Build Servers (case in point: even TFS requires fiddling around to work with Package Restore)


• Not everything lives in NuGet (native DLLs, for example)


• Package Restore often works unreliably, especically when updating back and forth between old and new versions of packages


• NuGet Package Restore is occasinally broken in some versions of Visual Studio and/or NuGet Integration Package


• All in all, it's too black-magick'y

Arguments against keeping them in a VCS and for NuGet Package Restore:

• Binary files can significantly increase repository size


• It's all very simple from the "end-developer" point of view


• It's Microsoft

In all honesty, I don't find them very appealing.

Am I missing something here? Are there any more solid arguments in favor of Package Restore?