Winsock 2 I/O Methods 5 Part 11

 

 

What do we have in this chapter 5 part 11?

1. Testing the Client-server Program

2. Completion Ports and Overlapped I/O

3. Per-handle Data and Per-I/O Operation Data

4. I/O Model Consideration

5. Client Development

6. Server Development

 

 

 

Completion Ports and Overlapped I/O

 

After associating a socket handle with a completion port, you can begin processing I/O requests by posting overlapped send and receive requests on the socket handle. You can now start to rely on the completion port for I/O completion notification. Basically, the completion port model takes advantage of the Windows overlapped I/O mechanism in which Winsock API calls such as WSASend() and WSARecv() return immediately when called. It is up to your application to retrieve the results of the calls at a later time through an OVERLAPPED structure. In the completion port model, this is accomplished by having one or more worker threads wait on the completion port using the GetQueuedCompletionStatus() function, which is defined as:

 

BOOL GetQueuedCompletionStatus(

    HANDLE CompletionPort,

    LPDWORD lpNumberOfBytesTransferred,

    PULONG_PTR lpCompletionKey,

    LPOVERLAPPED * lpOverlapped,

    DWORD dwMilliseconds

);

 

The CompletionPort parameter represents the completion port to wait on. The lpNumberOfBytesTransferred parameter receives the number of bytes transferred after a completed I/O operation, such as WSASend() or WSARecv(). The lpCompletionKey parameter returns per-handle data for the socket that was originally passed into the CreateIoCompletionPort() function. As we already mentioned, we recommend saving the socket handle in this key. The lpOverlapped parameter receives the WSAOVERLAPPED structure of the completed I/O operation. This is actually an important parameter because it can be used to retrieve per I/O-operation data, which we will describe shortly. The final parameter, dwMilliseconds, specifies the number of milliseconds that the caller is willing to wait for a completion packet to appear on the completion port. If you specify INFINITE, the call waits forever.

 

Per-handle Data and Per-I/O Operation Data

 

When a worker thread receives I/O completion notification from the GetQueuedCompletionStatus() API call, the lpCompletionKey and lpOverlapped parameters contain socket information that can be used to continue processing I/O on a socket through the completion port. Two types of important socket data are available through these parameters: per-handle data and per-I/O operation data.

The lpCompletionKey parameter contains what we call per-handle data because the data is related to a socket handle when a socket is first associated with the completion port. This is the data that is passed as the CompletionKey parameter of the CreateIoCompletionPort() API call. As we noted earlier, your application can pass any type of socket information through this parameter. Typically, applications will store the socket handle related to the I/O request here.

The lpOverlapped parameter contains an OVERLAPPED structure followed by what we call per-I/O operation data, which is anything that your worker thread will need to know when processing a completion packet (echo the data back, accept the connection, post another read, and so on). Per-I/O operation data is any number of bytes contained in a structure also containing an OVERLAPPED structure that you pass into a function that expects an OVERLAPPED structure. A simple way to make this work is to define a structure and place an OVERLAPPED structure as a field of the new structure. For example, we declare the following data structure to manage per-I/O operation data:

 

typedefstruct

{

    OVERLAPPED Overlapped;

   char       Buffer[DATA_BUFSIZE];

   int    BufferLen;

   int        OperationType;

} PER_IO_DATA;

 

This structure demonstrates some important data elements you might want to relate to an I/O operation, such as the type of I/O operation (a send or receive request) that just completed. In this structure, we consider the data buffer for the completed I/O operation to be useful. To call a Winsock API function that expects an OVERLAPPED structure, you dereference the OVERLAPPED element of your structure. For example:

 

PER_IO_OPERATION_DATA PerIoData;

WSABUF wbuf;

DWORD Bytes, Flags;

 

// Initialize wbuf ...

WSARecv(socket, &wbuf, 1, &Bytes, &Flags, &(PerIoData.Overlapped), NULL);

 

Later in the worker thread, GetQueuedCompletionStatus() returns with an overlapped structure and completion key. To retrieve the per-I/O data the macro CONTAINING_RECORD should be used. For example,

 

PER_IO_DATA  *PerIoData=NULL;

OVERLAPPED   *lpOverlapped=NULL;

 

ret = GetQueuedCompletionStatus(

         CompPortHandle,

         &Transferred,

         (PULONG_PTR)&CompletionKey,

         &lpOverlapped,            INFINITE);

 

// Check for successful return

PerIoData = CONTAINING_RECORD(lpOverlapped, PER_IO_DATA, Overlapped);

 

This macro should be used; otherwise, the OVERLAPPED member of the PER_IO_DATA structure would always have to appear first, which can be a dangerous assumption to make (especially with multiple developers working on the same code).

You can determine which operation was posted on this handle by using a field of the per-I/O structure to indicate the type of operation posted. In our example, the OperationType member would be set to indicate a read, write, etc., operation. One of the biggest benefits of per-I/O operation data is that it allows you to manage multiple I/O operations (such as read/write, multiple reads, and multiple writes) on the same handle. You might ask why you would want to post more than one I/O operation at a time on a socket. The answer is scalability. For example, if you have a multiple-processor machine with a worker thread using each processor, you could potentially have several processors sending and receiving data on a socket at the same time.

Before continuing, there is one other important aspect about Windows completion ports that needs to be stressed. All overlapped operations are guaranteed to be executed in the order that the application issued them. However, the completion notifications returned from a completion port are not guaranteed to be in that same order. That is, if an application posts two overlapped WSARecv() operations, one with a 10 KB buffer and the next with a 12 KB buffer, the 10 KB buffer is filled first, followed by the 12 KB buffer. The application's worker thread may receive notification from GetQueuedCompletionStatus() for the 12 KB WSARecv() before the completion event for the 10 KB operation. Of course, this is only an issue when multiple operations are posted on a socket. To complete this simple echo server sample, we need to supply a ServerWorkerThread() function. The following code outlines how to develop a worker thread routine that uses per-handle data and per-I/O operation data to service I/O requests.

 

 DWORD WINAPI ServerWorkerThread(LPVOID CompletionPortID)

{

    HANDLE CompletionPort = (HANDLE) CompletionPortID;

    DWORD BytesTransferred;

    LPOVERLAPPED Overlapped;

    LPPER_HANDLE_DATA PerHandleData;

    LPPER_IO_DATA PerIoData;

    DWORD SendBytes, RecvBytes;

    DWORD Flags;

 

   while(TRUE)

    {

       // Wait for I/O to complete on any socket

       // associated with the completion port   

        ret = GetQueuedCompletionStatus(CompletionPort, &BytesTransferred,(LPDWORD)&PerHandleData,

            (LPOVERLAPPED *) &PerIoData, INFINITE);

 

       // First check to see if an error has occurred

       // on the socket; if so, close the

       // socket and clean up the per-handle data

       // and per-I/O operation data associated with the socket

       if (BytesTransferred == 0 && (PerIoData->OperationType == RECV_POSTED ││ PerIoData->OperationType == SEND_POSTED))

        {

           // A zero BytesTransferred indicates that the

           // socket has been closed by the peer, so

           // you should close the socket. Note:

           // Per-handle data was used to reference the

           // socket associated with the I/O operation.

            closesocket(PerHandleData->Socket);

 

            GlobalFree(PerHandleData);

            GlobalFree(PerIoData);

           continue;

        }

 

 

       // Service the completed I/O request. You can

       // determine which I/O request has just

       // completed by looking at the OperationType

       // field contained in the per-I/O operation data.

        if (PerIoData->OperationType == RECV_POSTED)

        {

           // Do something with the received data in PerIoData->Buffer

        }

       // Post another WSASend or WSARecv operation.

       // As an example, we will post another WSARecv() I/O operation.

        Flags = 0;

       // Set up the per-I/O operation data for the next overlapped call

        ZeroMemory(&(PerIoData->Overlapped), sizeof(OVERLAPPED));

        PerIoData->DataBuf.len = DATA_BUFSIZE;

        PerIoData->DataBuf.buf = PerIoData->Buffer;

        PerIoData->OperationType = RECV_POSTED;

        WSARecv(PerHandleData->Socket, &(PerIoData->DataBuf), 1, &RecvBytes, &Flags, &(PerIoData->Overlapped), NULL);

    }

}

 

If an error has occurred for a given overlapped operation, GetQueuedCompletionStatus() will return FALSE. Because completion ports are a Windows I/O construct, if you call GetLastError() or WSAGetLastError(), the error code is likely to be a Windows error code and not a Winsock error code. To retrieve the equivalent Winsock error code, WSAGetOverlappedResult() can be called specifying the socket handle and WSAOVERLAPPED structure for the completed operation, after which WSAGetLastError() will return the translated Winsock error code.

One final detail not outlined in the last two examples we have presented is how to properly close an I/O completion port especially if you have one or more threads in progress performing I/O on several sockets. The main thing to avoid is freeing an OVERLAPPED structure when an overlapped I/O operation is in progress. The best way to prevent this is to call closesocket() on every socket handle any overlapped I/O operations pending will complete. Once all socket handles are closed, you need to terminate all worker threads on the completion port. This can be accomplished by sending a special completion packet to each worker thread using the PostQueuedCompletionStatus() function, which informs each thread to exit immediately. PostQueuedCompletionStatus() is defined as:

 

BOOL PostQueuedCompletionStatus(

    HANDLE CompletionPort,

    DWORD dwNumberOfBytesTransferred,

    ULONG_PTR dwCompletionKey,

    LPOVERLAPPED lpOverlapped

);

 

The CompletionPort parameter represents the completion port object to which you want to send a completion packet. The dwNumberOfBytesTransferred, dwCompletionKey, and lpOverlapped parameters each will allow you to specify a value that will be sent directly to the corresponding parameter of the GetQueuedCompletionStatus() function. Thus, when a worker thread receives the three passed parameters of GetQueuedCompletionStatus(), it can determine when it should exit based on a special value set in one of the three parameters. For example, you could pass the value 0 in the dwCompletionKey parameter, which a worker thread could interpret as an instruction to terminate. Once all the worker threads are closed, you can close the completion port using the CloseHandle() function and finally exit your program safely.

The completion port I/O model is by far the best in terms of performance and scalability. There are no limitations to the number of sockets that may be associated with a completion port and only a small number of threads are required to service the completed I/O.

 

I/O Model Consideration

 

By now you might be wondering how to choose the I/O model you should use when designing your application. As we've mentioned, each model has its strengths and weaknesses. All of the I/O models do require fairly complex programming compared with developing a simple blocking-mode application with many servicing threads. We offer the following suggestions for client and server development.

 

Client Development

 

When you are developing a client application that manages one or more sockets, we recommend using overlapped I/O or WSAEventSelect() over the other I/O models for performance reasons. But, if you are developing a Windows-based application that manages window messages, the WSAAsyncSelect model might be a better choice because WSAAsyncSelect() lends itself to the Windows message model, and your application is already set up for handling messages.

 

Server Development

 

When you are developing a server that processes several sockets at a given time, we recommend using overlapped I/O over the other I/O models for performance reasons. However, if you expect your server to service a large number of I/O requests at any given time, you should consider using the I/O completion port model for even better performance.