Object creation: when should I expose a factory vs wrapping class?

I am having some problems trying to figure out when to use a factory or to wrapper class. This question is slightly geared towards C# I guess, so I'm not sure if this is the correct place to ask.

Say there is a library that exposes an interface called IStackExchangeClient, and multiple differing implementations of that interface (private implementations that clients will not know about), HttpStackExchangeClient, UdpStackExchangeClient.

We can allow the object creation of the interface through a factory, like StackExchangeClientFactory.Create(), or we can wrap it up in another class like StackExchangeClient that internally does the same thing, but simply proxies.

In the second instance, clients can write new StackExchangeClient() which feels much more natural to me. So my question is, when do I choose one or the other? What are some of the considerations that I should be taking in?

I've thought hard about it and I can't choose between one style or the other, but clearly some parts of the .NET framework use factories, and some parts don't, even when there are multiple underlying implementations of its interface.

Thank you!