Task based timeout, activating slower than expected

I have an HttpMessage, which at some point expects a result. I want the user to be able to await a task which returns ether after the response has been set, or after the timeout has elapsed. When running my unit tests, it works but only if i set the timeout much sooner than the delayed setting of the response.

Why is this happening, and is there anything I can do to tighten up the timing.

*Note the unit test below works, but if instead set the delay to 500ms, it fails

HttpMessage code:

public async Task<bool> ContinueWhenResponseRecieved(int Timeout)
        {
            NotifyTaskCompletionSource = new TaskCompletionSource<bool>();
            bool result = true;
            //If we have already recieved a response return true;
            if (this._response != null)
            {
                return result;
            }
            //If Timeout is not 0 then start a task that after the timeout period will set the completion source to true no matter what
            if (Timeout != 0) 
            {
                TaskEx.Run(() =>
                {
                    TaskEx.Delay(Timeout).Wait();
                    result = false;
                    try
                    {
                        NotifyTaskCompletionSource.SetResult(true);
                    }
                    catch (Exception ex) { } //ignore race condition issues
                });
            }
            await NotifyTaskCompletionSource.Task;
            return result;
        }
        public override IHttpMessage setResponse(HttpResponse _response)
        {
            if (NotifyTaskCompletionSource != null)
            {
                try
                {
                    NotifyTaskCompletionSource.SetResult(true);
                }catch(Exception ex){} //Ignore race condtion issues
            }
            return base.setResponse(_response);
        }

Unit Test:

  [TestMethod]
    public void TestNotifiableTimeOuts()
    {
        var msg = new NotifiableHttpMessage();
        TaskEx.Run(() =>
        {
            TaskEx.Delay(10000).Wait();
            msg.setResponse(new Messages.HttpResponse());
        });
        Task<bool> tsk = msg.ContinueWhenResponseRecieved(50);
        tsk.Wait();

        Assert.IsFalse(tsk.Result);
    }