How to free server side resources in RMI call?

Our application is a client-server based application that I have converted from .NET remoting to ZMQ. We have two scenarios where a large image object is converted to a byte array in order to be serialized and sent to the client. In order to reduce memory issues (Large Object Heap) we have a pool of byte arrays that are used for this process, thus eliminating allocating and freeing them constantly.

The problem I have is that I do not know how to mark the byte arrays as "free" in the pool after returning the array in an RMI-like call.

public byte[] GetImageAreaGrey(int imageId)
    {
        var image = FindImageById(imageId);
        byte[] byteHolder = BytePool.GetFreeArray();
        ByteConverter.ConvertImageToBytes(image, byteHolder)
        return byteHolder ; // byteHolder will be serialized in the communications code using JSON.NET
        // How do I return byteHolder to the pool so that is available for future use?
    }

The application is written in C# with .NET 4.5 run time and we are trying to prevent fragmentation of the Large Object Heap.

edit: clarified that the raw byte[] is being returned.