Fortunately, Microsoft® .NET Framework interop opens a channel between managed and unmanaged code, and marshaling plays a very important role in that connection in that it allows for data exchange between the two (see Figure 1). There are many factors that affect the way the CLR marshals data between the unmanaged and managed worlds, including attributes such as [MarshalAs], [StructLayout], [InAttribute], and [OutAttribute] as well as language keywords such as out and ref in C#.
Because of the sheer number of these factors, it can be a challenge to marshal correctly, which requires you to understand many details in both unmanaged and managed code. In this column, we'll cover the basic yet confusing topics you'll encounter in your everyday marshaling attempts. We won't cover custom marshaling, marshaling complex structures, or other advanced topics, but with a solid understanding of the basics, you'll be ready to begin tackling those as well.
The first marshaling topic we'll discuss concerns the use of InAttribute and OutAttribute, attribute types located in the System.Runtime.InteropServices namespace. (When applying these attributes in your code, C# and Visual Basic® allow you to use the abbreviated forms [In] and [Out], but we will stick to the full names to avoid confusion.)
When applying to method parameters and return values, these attributes control marshaling direction, so they're known as directional attributes. [InAttribute] tells the CLR to marshal data from the caller to the callee at the beginning of the call, while [OutAttribute] tells the CLR to marshal back from the callee to the caller upon return. Both the caller and the callee can be either unmanaged or managed code. For example, in a P/Invoke call, managed code is calling unmanaged code. However, in a reverse P/Invoke, unmanaged code could call managed code through a function pointer.
There are four possible combinations in which [InAttribute] and [OutAttribute] can be used: [InAttribute] only, [OutAttribute] only, [InAttribute, OutAttribute], and neither. When neither attribute is specified, you are telling the CLR to figure out directional attributes by itself, with [InAttribute] usually applied by default. However, in the case of the StringBuilder class, both [InAttribute] and [OutAttribute] are applied when neither attribute is specified. (See the later section on StringBuilder for details.) Also, the use of out and ref keywords in C# could change the attributes that are applied, as shown in Figure 2. Note that if no keyword is specified for a parameter, it means that it is an input parameter by default.
Consider the code in Figure 3. There are three native C++ functions, and they are all making the same change to arg. Also, note that the use of strcpy for string manipulation is for illustrative purposes only—production code should instead use the safe versions of these functions, which can be found at msdn.microsoft.com/msdnmag/issues/05/05/SafeCandC.
The only thing that is different is how we will invoke them, using directional attributes in the P/Invoke signatures, as shown in the following C# code:
[DllImport(@"MarshalLib.dll")] public static extern void Func_In_Attribute([In]char[] arg); [DllImport(@"MarshalLib.dll")] public static extern void Func_Out_Attribute([Out]char[] arg); [DllImport(@"MarshalLib.dll")] public static extern void Func_InOut_Attribute([In, Out]char[] arg);
If you call these functions from managed code by P/Invoke and pass in "Old" as a char array to the functions, you'll get the following output (abridged for demonstration purposes):
Before Func_In_Attribute: arg = Old Inside Func_In_Attribute: arg = Old After Func_In_Attribute: arg = Old Before Func_Out_Attribute: arg = Old Inside Func_Out_Attribute: arg = After Func_Out_Attribute: arg = New Before Func_InOut_Attribute: arg = Old Inside Func_InOut_Attribute: arg = Old After Func_InOut_Attribute: arg = New
Let's look at the results more closely. In Func_In_Attribute, the original value is passed in, but the change made inside Func_In_Attribute is not propagated back. In Func_Out_Attribute, the original value is not passed in and the change made inside Func_Out_Attribute is propagated back. In Func_InOut_Attribute, the original value is passed in and the change made inside Func_Out_Attribute is propagated back. However, things could be totally different if you make a slight change. This time, let's change the native function to use Unicode, as shown below:
MARSHALLIB_API void __stdcall Func_Out_Attribute_Unicode(wchar_t *arg)
{
wprintf(L"Inside Func_Out_Attribute_Unicode: arg = %s/n", arg);
printf("Inside Func_Out_Attribute_Unicode: strcpy(arg, /"New/")/n");
wcscpy(arg, L"New");
}
Here we declare the C# function, apply only [OutAttribute], and change CharSet to CharSet.Unicode:
[DllImport(@"MarshalLib.dll", CharSet=CharSet.Unicode)] public static extern void Func_Out_Attribute_Unicode([Out]char[] arg);
Here's the output:
Before Func_Out_Attribute_Unicode: arg = Old Inside Func_Out_Attribute_Unicode: arg = Old After Func_Out_Attribute_Unicode: arg = New
Interestingly, the original value is passed even though there is no [InAttribute]. The [DllImportAttribute] tells the CLR to marshal Unicode, and, because char type in the CLR is also Unicode, the CLR sees an opportunity to optimize the marshaling process by pinning the char array and passing the address of the char directly. (You'll see a detailed discussion of copying and pinning later.) However, this doesn't mean you should rely on this behavior. Instead, the correct marshaling directional attributes should always be used when not relying on the CLR's default marshaling behavior. A typical example of this default behavior is the case of an int argument; specifying [InAttribute] int arg isn't necessary.
There are cases when [OutAttribute] will be ignored. For example, [OutAttribute]int doesn't make any sense, so the [OutAttribute] is simply ignored by the CLR. The same is true of [OutAttribute] string because string is immutable.
Interface definition (IDL) files also have [in] and [out] attributes, which can be considered the same as [InAttribute] and [OutAttribute] in the CLR.
Previously, we've shown that C# out and ref keywords can be directly mapped to [InAttribute] and [OutAttribute]. As a matter of fact, out and ref can also change what data type the CLR will marshal into or from. Passing data as out or ref is the same as passing by reference. If you examine the corresponding function signature in intermediate language (IL) using ILDASM, you can see there is an ampersand character (&) next to the type, which means that the argument should be passed by reference. When passing by reference, an extra level of indirection will be added by the CLR. Figure 4 shows some examples.
Let's summarize what we discussed for out and ref in the table shown in Figure 5.
Please note that when passing by reference, if no directional attributes are specified, the CLR will apply [InAttribute] and [OutAttribute] automatically, which is why there is only "string &" in the Microsoft Intermediate Language (MSIL) signature in Figure 4. If any of these attributes are specified, the CLR will follow them instead of the default behavior, as shown in this example:
public static extern void
PassPointerToComplexStructure(
[In]ref ComplexStructure
pStructure);
The signature above will override the default directional behavior of ref, making it [InAttribute]-only. In this particular case, if you are doing a P/Invoke, a pointer to ComplexStructure (which is a value type) is being passed from the CLR side to the native side, but the callee can't make any changes visible to the ComplexStructure pointed to by pStructure pointer. Figure 6 shows other examples of directional attributes and keyword combinations.
So far we have only been discussing arguments. What about values returned from functions? The CLR automatically treats a return value as if it is a normal argument using the [OutAttribute]. Also, the CLR can make the transformation to the function signature, a process controlled by the PreserveSigAttribute. If [PreserveSigAttribute] is set to false when applied to a P/Invoke signature, the CLR will map HRESULT return values to managed exceptions, and it will map [out, retval] parameters to the function's return value. So the following managed function signature
public static string extern GetString(int id);
would become the unmanaged signature:
HRESULT GetString([in]int id, [out, retval] char **pszString);
If [PreserveSigAttribute] is set to true (the default for P/Invoke), this transformation won't happen. Note that with COM functions, [PreserveSigAttribute] is usually set to false by default, though there are a number of ways to change that. For details, please consult the MSDN® documentation for TlbExp.exe and TlbImp.exe.
The CLR marshaler has built-in knowledge of the StringBuilder type and treats it differently from other types. By default, StringBuilder is passed as [InAttribute, OutAttribute]. StringBuilder is special because it has a Capacity property that can determine the size of the required buffer at run time, and it can be changed dynamically. Therefore, during the marshaling process, the CLR can pin StringBuilder, directly pass the address of internal buffer used in StringBuilder, and allow the contents of this buffer to be changed by native code in place.
To take full advantage of StringBuilder, you'll need to follow all of these rules:
When the CLR performs data marshaling, it has two options: copying and pinning (see msdn2.microsoft.com/23acw07k).
By default, the CLR will make a copy that will be used during marshaling. For example, if managed code is passing a string as an ANSI C-String to unmanaged code, the CLR will make a copy of the string, convert it to ANSI, and then pass the temporary's pointer to unmanaged code. That copying can be quite slow and can create performance problems.
In certain cases, the CLR is able to optimize the marshaling process by directly pinning the managed object in the Garbage Collector (GC) heap so that it cannot be relocated during the call. The pointer to the managed object (or to somewhere inside the managed object) will be passed directly to unmanaged code.
Pinning is performed when all of the following conditions are met: First, managed code must be calling native code, not the other way around. Second, the type must be blittable or must be capable of becoming blittable under certain conditions. Third, you're not passing by reference (out or ref), and fourth, the caller and callee are in the same thread context or apartment.
The second rule deserves further explanation. A blittable type is one that has a common representation in both managed and unmanaged memory. As such, blittable types do not require conversion when marshaling. A typical example of a type that is non-blittable, but can become blittable, is the char type. By default, it is non-blittable since it can be mapped to either Unicode or ANSI. However, because char is always Unicode in the CLR, it does become blittable when you specify [DllImportAttribute(CharSet= Unicode)] or [MarshalAsAttribute(UnmanagedType.LPWSTR)]. In the following example, arg can be pinned in PassUnicodeString, but it cannot be pinned in PassAnsiString:
[DllImport(@"MarshalLib.dll", CharSet = CharSet.Unicode)] public static extern string PassUnicodeString(string arg); [DllImport(@"MarshalLib.dll", CharSet = CharSet.Ansi)] public static extern string PassAnsiString(string arg);
During a function call, a function can make two kinds of changes to its arguments: a reference change or an in-place change. A reference change involves changing where a pointer points; if the pointer already points to a piece of allocated memory, that memory may first need to be freed before the pointer to it is lost. An in-place change involves changing the memory at the location pointed to by the reference.
Which of these changes is made depends on the type of the argument and, most importantly, the contract between the callee and caller. However, since the CLR cannot figure out the contract automatically, it has to rely on common knowledge about types, as illustrated in Figure 7.
As previously discussed, only reference types have two levels of indirection when passing by reference (there are a few exceptions, however, such as "[MarshalAs(UnmanagedType.LPStruct)]ref Guid"), so only the pointer or reference to a reference type can be changed, as shown in Figure 8.
You don't need to worry about memory ownership for an in-place change because the caller has allocated memory for the callee and the caller owns the memory. Let's take "[OutAttribute] StringBuilder" as an example here. The corresponding native type is char * (assuming ANSI), since we are not passing by reference. Data is marshaled out, not in. Memory is allocated by the caller (in this case the CLR). The size of the memory is determined by the capacity of the StringBuilder object. The callee doesn't need to concern itself with the memory.
To change the string, the callee will make the change directly to the memory itself. However, when making a reference change, it is critical to recognize who owns which memory, otherwise any number of unexpected outcomes can occur. Regarding ownership issues, the CLR follows COM-style conventions:
In the interop world, caller/callee becomes CLR/native code. The rules above imply that in the unpinned case, if when in native code you receive a pointer to a block of memory passed to you as [out] from the CLR, you need to free it. On the other hand, if the CLR receives a pointer that is passed as [out] from native code, the CLR needs to free it. Clearly, in the first case, native code needs to do the de-allocation and in the second case, managed code needs to do de-allocation.
Since this involves memory allocation and de-allocation, the biggest problem is what function to use. There are many to choose from: HeapAlloc/HeapFree, malloc/free, new/delete, and so on. However, since the CLR uses CoTaskMemAlloc/CoTaskMemFree in the non-BSTR case and SysStringAlloc/SysStringAllocByteLen/SysStringFree in the BSTR case, you'll have to use those functions. Otherwise, it is likely that you'll get a memory leak or a crash in certain versions of Windows®. We have seen cases where malloc'd memory was passed to the CLR and the program didn't crash in Windows XP but did crash in Windows Vista®.
Besides those functions, a system-implemented IMalloc interface returned from CoGetMalloc also works fine because internally they are using the same heap. However, it is best to always stick with CoTaskMemAlloc/CoTaskMemFree and SysStringAlloc/ SysStringAllocByteLen/SysStringFree, since CoGetMalloc is subject to future change.
Let's look at an example. The GetAnsiStringFromNativeCode takes a char ** argument as [in, out] and returns a char * as [out, retval]. For the char ** argument, it can choose to call CoTaskMemFree to free the memory that is allocated by the CLR, then allocate new memory by using CoTaskMemAlloc and overwrite the pointer with a new memory pointer. Later, the CLR will free the memory and create a copy for the managed string. As for the return value, it only needs to allocate a new piece of memory by using CoTaskMemAlloc and return it to the caller. After return, the newly allocated memory is now owned by the CLR. The CLR will first create a new managed string from it and then call CoTaskMemFree to free it.
Let's consider the first option (see Figure 9). The corresponding C# function declaration is as follows:
class Lib
{
[DllImport(@"MarshalLib.dll", CharSet= CharSet.Ansi)]
public static extern string GetAnsiStringFromNativeCode(
ref string inOutString);
}
When the following C# code makes a call to GetAnsiStringFromNativeCode
string argStr = "Before";
Console.WriteLine("Before GetAnsiStringFromNativeCode : argStr = /"" +
argStr + "/"");
string retStr = Lib.GetAnsiStringFromNativeCode(ref argStr);
Console.WriteLine("AnsiStringFromNativeCode() returns /"" + retStr +
"/"" );
Console.WriteLine("After GetAnsiStringFromNativeCode : argStr = /"" +
argStr + "/"");
the output is:
Before GetAnsiStringFromNativeCode : argStr = "Before" Inside GetAnsiStringFromNativeCode: *arg = Before Inside GetAnsiStringFromNativeCode: CoTaskMemFree(*arg); *arg = CoTaskMemAlloc(100); strcpy(*arg, "Changed") AnsiStringFromNativeCode() returns "Returned String From Native Code" After GetAnsiStringFromNativeCode : argStr = "Changed"
If the native function you are going to call doesn't follow this convention, you'll have to do the marshaling by yourself to avoid memory corruption. This could easily happen because the function for an unmanaged function could return whatever it wants; it can return the same piece of memory every time or return a new block of memory allocated by malloc/new, and so forth, again depending on the contract.
Besides memory allocation, the size of the memory being passed in or out is also very important. As discussed in the StringBuilder case, it is very important to change the Capacity property so that the CLR can allocate a piece of memory big enough to hold the results. In addition, marshaling a string as [InAttribute, OutAttribute] (without out or ref and any other attribute) is a bad idea because you don't know whether the string will be big enough. You can use SizeParamIndex and SizeConst fields in MarshalAsAttribute to specify the size of the buffer. However, these attributes cannot be used when passing by reference.
The CLR allows you to pass a delegate to the unmanaged world so that it can call the delegate as an unmanaged function pointer. In fact, what's happening is that the CLR creates a thunk, which forwards the calls from native code to the actual delegate, then to the real function (see Figure 10).
Usually, you won't have to worry about the lifetime of delegates. Whenever you are passing a delegate to unmanaged code, the CLR will make sure the delegate is alive during the call.
However, if the native code keeps a copy of the pointer beyond the span of the call and intends to call back through that pointer later, you might need to use GCHandle to explicitly prevent the garbage collector from collecting the delegate. We must warn you that a pinned GCHandle could have a significantly negative impact on program performance. Fortunately, in this case, you don't need to allocate a pinned GC handle, because the thunk is allocated in the unmanaged heap and is referencing the delegate indirectly through a reference known to the GC. Therefore, it is not possible for the thunk to move around, and native code should always be able to call the delegate through the unmanaged pointer if the delegate itself is alive.
Marshal.GetFunctionPointerForDelegate can convert a delegate to a function pointer, but it doesn't do anything to guarantee the lifetime of the delegate. Consider the following function declaration:
public delegate void PrintInteger(int n);
[DllImport(@"MarshalLib.dll", EntryPoint="CallDelegate")]
public static extern void CallDelegateDirectly(
IntPtr printIntegerProc);
If you call Marshal.GetFunctionPointerForDelegate for it and store the returned IntPtr, then pass the IntPtr to the function you are going to call, like so:
IntPtr printIntegerCallback = Marshal.GetFunctionPointerForDelegate(
new Lib.PrintInteger(MyPrintInteger));
GC.Collect();
GC.WaitForPendingFinalizers();
GC.Collect();
CallDelegateDirectly(printIntegerCallback);
It is possible that the delegate will be collected before you call CallDelegateDirectly, and you will get an MDA error that CallbackOnCollectedDelegate was detected. To fix that, you can either store a reference to the delegate in memory or allocate a GC handle.
If native code returns an unmanaged function pointer to the CLR, it is the responsibility of native code to keep the actual function code around. Usually this isn't a problem unless the code is in a dynamically loaded DLL or generated on the fly.
Understanding and remembering all the attributes and rules described so far may seem a bit daunting. After all, most developers of managed code just need to be able to quickly figure out the P/Invoke signature for a Win32® API function, paste it in their code, and be done with it. That's where the P/Invoke Interop Assistant (available at the MSDN Magazine Web site) can help. This tool can efficiently assist with conversions from C++ to managed P/Invoke signatures as well as those in the opposite direction. It even comes with a database of Win32 functions, data types, and constants, so the common task of adding a Win32 P/Invoke to your C# or Visual Basic source file is made very easy. The tool package includes two command-line tools, SigImp and SigExp, which can be used for batch file processing. A GUI tool is also found in the package, which includes the functionalities of both tools.
The GUI tool is handy for doing simple conversions. It contains three tabs: SigExp, SigImp Search, and SigImp Translate Snippet.
SigExp converts managed signature to an unmanaged signature. It reflects over managed assemblies to find all P/Invoke declarations and COM imported types. From this input it produces the corresponding native C signatures (see Figure 11).
SigImp Search and SigImp Translate Snippet convert unmanaged signatures to managed signatures. They generate managed signatures and definitions in either C# or Visual Basic from native types, functions, constants, and snippets of manually entered native function signatures.
SigImp Search allows users to choose the managed code language in which they want code generated and then select a native type, procedure, or constant to perform the generation from. The tool will display a list of supported types, methods, and constants collected from Windows SDK header files (see Figure 12).
SigImp Translate Snippet allows users to write their own native code snippet in the tool. The tool will then generate and display the managed code equivalent in the main window, as you see in Figure 13.
For details of the GUI tool or the command-line tools in the P/Invoke Interop Assistant, please refer to the documentation packed with the tool.
As you know, marshaling is a complex topic, and there are many techniques you can use to change the marshaling process to adapt to your own needs. We suggest you try some of the ideas presented here. They're sure to help you find your way through the maze we know as marshaling.