<?xml version="1.0" encoding="UTF-8"?>
<rss version="2.0"
	xmlns:content="http://purl.org/rss/1.0/modules/content/"
	xmlns:wfw="http://wellformedweb.org/CommentAPI/"
	xmlns:dc="http://purl.org/dc/elements/1.1/"
	xmlns:atom="http://www.w3.org/2005/Atom"
	xmlns:sy="http://purl.org/rss/1.0/modules/syndication/"
	xmlns:slash="http://purl.org/rss/1.0/modules/slash/"
	>

<channel>
	<title>Basildon Coder &#187; Data Structures</title>
	<atom:link href="http://basildoncoder.com/blog/category/development/data-structures/feed/" rel="self" type="application/rss+xml" />
	<link>http://basildoncoder.com/blog</link>
	<description>Incoherent and disjointed opinionated drivel from somewhere near London</description>
	<lastBuildDate>Fri, 11 Dec 2009 13:40:35 +0000</lastBuildDate>
	<language>en</language>
	<sy:updatePeriod>hourly</sy:updatePeriod>
	<sy:updateFrequency>1</sy:updateFrequency>
	<generator>http://wordpress.org/?v=3.0</generator>
		<item>
		<title>Marshalling a Variable-Length Array From Unmanaged Code In C#</title>
		<link>http://basildoncoder.com/blog/2009/03/31/marshalling-a-variable-length-array-from-unmanaged-code-in-c/</link>
		<comments>http://basildoncoder.com/blog/2009/03/31/marshalling-a-variable-length-array-from-unmanaged-code-in-c/#comments</comments>
		<pubDate>Tue, 31 Mar 2009 19:03:38 +0000</pubDate>
		<dc:creator>russ</dc:creator>
				<category><![CDATA[.Net]]></category>
		<category><![CDATA[Coding]]></category>
		<category><![CDATA[Data Structures]]></category>
		<category><![CDATA[Software Engineering]]></category>

		<guid isPermaLink="false">http://basildoncoder.com/blog/2009/03/31/marshalling-a-variable-length-array-from-unmanaged-code-in-c/</guid>
		<description><![CDATA[I recently spent time working on some C# code to interact with a simple DNS-SD system. This requires using DNS TXT records, which are not supported in the System.Net.Dns class. After a few google searches failed to turn up a pure .Net client library that met my needs, I settled on an approach based around [...]]]></description>
			<content:encoded><![CDATA[<p>I recently spent time working on some C# code to interact with a simple <a href="http://files.dns-sd.org/draft-cheshire-dnsext-dns-sd.txt">DNS-SD</a> system. This requires using <a href="http://en.wikipedia.org/wiki/List_of_DNS_record_types">DNS TXT records</a>, which are not supported in the <a href="http://msdn.microsoft.com/en-us/library/system.net.dns.aspx">System.Net.Dns</a> class. After a few google searches failed to turn up a pure .Net client library that met my needs, I settled on an approach based around p/invoking the Win32 <a href="http://msdn.microsoft.com/en-us/library/ms682016(VS.85).aspx">DnsQuery</a> function.</p>
<p>And quickly ran into problems.</p>
<p>For DNS TXT records, DnsQuery returns a <a href="http://msdn.microsoft.com/en-us/library/ms682109(VS.85).aspx">DNS_TXT_DATA</a> structure in the Data field of the <a href="http://msdn.microsoft.com/en-us/library/ms682082(VS.85).aspx">DNS_RECORD</a> structure. DNS_TXT_DATA is declared like this:</p>

<div class="wp_syntax"><div class="code"><pre class="c" style="font-family:monospace;"><span style="color: #993333;">typedef</span> <span style="color: #993333;">struct</span> <span style="color: #009900;">&#123;</span>
    DWORD dwStringCount<span style="color: #339933;">;</span>
    PWSTR pStringArray<span style="color: #009900;">&#91;</span><span style="color: #0000dd;">1</span><span style="color: #009900;">&#93;</span><span style="color: #339933;">;</span>
<span style="color: #009900;">&#125;</span> DNS_TXT_DATA<span style="color: #339933;">,</span>
    <span style="color: #339933;">*</span>PDNS_TXT_DATA<span style="color: #339933;">;</span></pre></div></div>

<p>Using the very handy <a href="http://clrinterop.codeplex.com/">P/Invoke Interop Assistant</a>, we see that this struct can be represented like this in managed code:</p>

<div class="wp_syntax"><div class="code"><pre class="csharp" style="font-family:monospace;"><span style="color: #000000;">&#91;</span>StructLayout<span style="color: #000000;">&#40;</span>LayoutKind.<span style="color: #0000FF;">Sequential</span><span style="color: #000000;">&#41;</span><span style="color: #000000;">&#93;</span>
<span style="color: #0600FF;">public</span> <span style="color: #FF0000;">struct</span> DNS_TXT_DATA <span style="color: #000000;">&#123;</span>
&nbsp;
    <span style="color: #008080; font-style: italic;">/// DWORD-&gt;unsigned int</span>
    <span style="color: #0600FF;">public</span> <span style="color: #FF0000;">uint</span> dwStringCount<span style="color: #008000;">;</span>
&nbsp;
    <span style="color: #008080; font-style: italic;">/// PWSTR[1]</span>
    <span style="color: #000000;">&#91;</span>MarshalAs<span style="color: #000000;">&#40;</span>UnmanagedType.<span style="color: #0000FF;">ByValArray</span>,
            SizeConst<span style="color: #008000;">=</span><span style="color: #FF0000;">1</span>,
            ArraySubType<span style="color: #008000;">=</span>UnmanagedType.<span style="color: #0000FF;">SysUInt</span><span style="color: #000000;">&#41;</span><span style="color: #000000;">&#93;</span>
    <span style="color: #0600FF;">public</span> IntPtr<span style="color: #000000;">&#91;</span><span style="color: #000000;">&#93;</span> pStringArray<span style="color: #008000;">;</span>
<span style="color: #000000;">&#125;</span></pre></div></div>

<p>There is a problem with pStringArray, unfortunately. The <a href="http://msdn.microsoft.com/en-us/library/system.runtime.interopservices.marshal.aspx">System.Runtime.InteropServices.Marshal</a> class cannot marshal a variable length array, as it needs to know in advance how big the array is in order to allocate memory. That&#8217;s why the managed structure needs SizeConst specified in the MarshalAs attribute.</p>
<p>However, if the DNS TXT record data contains multiple quoted strings separated by whitespace, DnsQuery will return a structure with a variable number of elements in pStringArray. Since SizeConst is set at compile-time, when we marshal this into the managed struct defined above, we only get the first element in our single-element array. Rats.</p>
<p>More googling turned up very little info on dealing with this, though I found indications that others had run into the same problem without finding a satisfactory conclusion. DnsQuery is not the only Win32 function that returns variable-length arrays, and p/invoking any of the others has the same issue.</p>
<p>Simply declaring SizeConst to be bigger than we need &#8211; &#8220;hey, I know I&#8217;ll never get more than 10 or so strings back, so why not declare SizeConst to be 128?&#8221; &#8211; is inelegant (hardcoded upper limits, ugh) and doesn&#8217;t work properly anyway. Since the struct layout is sequential the marshaller will copy over (e.g.) 128*sizeof(IntPtr) sequential bytes (a total of 512 bytes, in this case). That much memory was never allocated on the unmanaged side, so we end up with a load of junk in the tail of pStringArray, and more often than not the marshaller chokes on this junk and throws an AccessViolationException. Fun.</p>
<p>There IS a way to get round the problem, though. I&#8217;m not sure it&#8217;s the best way, but it works and seems stable, so I thought I&#8217;d throw it out there in case anyone else can use it (or maybe explain to me why it&#8217;s an unsafe stupid thing to do&#8230;)</p>
<p>Basically, since we&#8217;re dealing with sequential memory, we can use Marshal.PtrToStructure to marshal the DNS_TXT_DATA structure as defined above, then use pointer arithmetic to gain access to any further data that needs marshalling.</p>
<p>Pointer arithmetic? Oh yes, even in the safe and secure world of managed code it&#8217;s sometimes still necessary to get our hands dirty, and situations like this illustrate that it will always be valuable to have some hard-earned Assembly/C/C++ war wounds.</p>
<p>So, assuming we have valid p/invoke declarations and data structures (I&#8217;ve included a complete source program below), DnsQuery is called like so:</p>

<div class="wp_syntax"><div class="code"><pre class="csharp" style="font-family:monospace;">var pServers <span style="color: #008000;">=</span> IntPtr.<span style="color: #0000FF;">Zero</span><span style="color: #008000;">;</span>
var ppQueryResultsSet <span style="color: #008000;">=</span> IntPtr.<span style="color: #0000FF;">Zero</span><span style="color: #008000;">;</span>
var ret <span style="color: #008000;">=</span> DnsQuery<span style="color: #000000;">&#40;</span>domain,
        DnsRecordType.<span style="color: #0000FF;">TEXT</span>,
        DnsQueryType.<span style="color: #0000FF;">STANDARD</span>,
        pServers,
        <span style="color: #0600FF;">ref</span> ppQueryResultsSet,
        IntPtr.<span style="color: #0000FF;">Zero</span><span style="color: #000000;">&#41;</span><span style="color: #008000;">;</span>
<span style="color: #0600FF;">if</span> <span style="color: #000000;">&#40;</span>ret <span style="color: #008000;">!=</span> <span style="color: #FF0000;">0</span><span style="color: #000000;">&#41;</span>
    <span style="color: #0600FF;">throw</span> <span style="color: #008000;">new</span> ApplicationException<span style="color: #000000;">&#40;</span><span style="color: #666666;">&quot;DnsQuery failed: &quot;</span> <span style="color: #008000;">+</span> ret<span style="color: #000000;">&#41;</span><span style="color: #008000;">;</span></pre></div></div>

<p>If we examine the memory location of ppQueryResultsSet (Ctrl-Alt-M,1 or Debug-&gt;Windows-&gt;Memory-&gt;Memory1 in Visual Studio) we&#8217;ll see something like the following (actual address locations may vary &#8211; just copy the int value of ppQueryResultsSet to the Address bar of the memory window):</p>
<pre>0x049E0878  00 00 00 00  ....
0x049E087C  b8 09 9e 04  ¸.ž.
0x049E0880  10 00 20 00  .. .
0x049E0884  19 30 00 00  .0..
0x049E0888  00 00 00 00  ....
0x049E088C  00 00 00 00  ....
0x049E0890  06 00 00 00  ....
0x049E0894  b8 08 9e 04  ¸.ž.
0x049E0898  d8 08 9e 04  Ø.ž.
0x049E089C  f8 08 9e 04  ø.ž.
0x049E08A0  28 09 9e 04  (.ž.
0x049E08A4  68 09 9e 04  h.ž.
0x049E08A8  88 09 9e 04  ˆ.ž.</pre>
<p>I&#8217;ve set the column size to 4 here, as most of the values we are dealing with are 4 bytes in size. This effectively shows one value per line.</p>
<p>The first 6 rows (24 bytes) correspond to the DNS_RECORD structure up until (but not including) the DNS_TXT_DATA structure in DNS_RECORD&#8217;s Data union. We can marshal this first structure without problem:</p>

<div class="wp_syntax"><div class="code"><pre class="csharp" style="font-family:monospace;">var dnsRecord <span style="color: #008000;">=</span> <span style="color: #000000;">&#40;</span>DnsRecord<span style="color: #000000;">&#41;</span> Marshal.<span style="color: #0000FF;">PtrToStructure</span><span style="color: #000000;">&#40;</span>
        ppQueryResultsSet, <span style="color: #008000;">typeof</span> <span style="color: #000000;">&#40;</span>DnsRecord<span style="color: #000000;">&#41;</span><span style="color: #000000;">&#41;</span><span style="color: #008000;">;</span></pre></div></div>

<p>The DNS_TXT_DATA structure starts at address 0x049E0890 in my example. Having already marshalled the DNS_RECORD structure, now I want a pointer to the DNS_TXT_DATA structure. I can do this by creating a new pointer at the address of ppQueryResultsSet plus 24 bytes, and marshalling again:</p>

<div class="wp_syntax"><div class="code"><pre class="csharp" style="font-family:monospace;">var ptr <span style="color: #008000;">=</span> <span style="color: #008000;">new</span> IntPtr<span style="color: #000000;">&#40;</span>
        ppQueryResultsSet.<span style="color: #0000FF;">ToInt32</span><span style="color: #000000;">&#40;</span><span style="color: #000000;">&#41;</span> <span style="color: #008000;">+</span> Marshal.<span style="color: #008000;">SizeOf</span><span style="color: #000000;">&#40;</span>dnsRecord<span style="color: #000000;">&#41;</span><span style="color: #000000;">&#41;</span><span style="color: #008000;">;</span>
var txtData <span style="color: #008000;">=</span> <span style="color: #000000;">&#40;</span>DNS_TXT_DATA<span style="color: #000000;">&#41;</span> Marshal.<span style="color: #0000FF;">PtrToStructure</span><span style="color: #000000;">&#40;</span>
        ptr, <span style="color: #008000;">typeof</span> <span style="color: #000000;">&#40;</span>DNS_TXT_DATA<span style="color: #000000;">&#41;</span><span style="color: #000000;">&#41;</span><span style="color: #008000;">;</span></pre></div></div>

<p>Because of the definition of DNS_TXT_DATA, this only marshals 8 bytes &#8211; 4 bytes for dwStringCount, and 4 bytes for the single element in pStringArray (an IntPtr). Since we know the memory is sequential, however, this gives us everything we need &#8211; we now know how many strings have been received (6 in this case, as indicated at 0x049E0890), and the location of the pointer to the first string (0x049E0894).</p>
<p>With this info, we can marshal all the pointers into an array with a length of dwStringCount:</p>

<div class="wp_syntax"><div class="code"><pre class="csharp" style="font-family:monospace;">ptr <span style="color: #008000;">=</span> <span style="color: #008000;">new</span> IntPtr<span style="color: #000000;">&#40;</span>ptr.<span style="color: #0000FF;">ToInt32</span><span style="color: #000000;">&#40;</span><span style="color: #000000;">&#41;</span> <span style="color: #008000;">+</span> <span style="color: #008000;">sizeof</span> <span style="color: #000000;">&#40;</span><span style="color: #FF0000;">uint</span><span style="color: #000000;">&#41;</span><span style="color: #000000;">&#41;</span><span style="color: #008000;">;</span> <span style="color: #008080; font-style: italic;">// move to first</span>
var ptrs <span style="color: #008000;">=</span> <span style="color: #008000;">new</span> IntPtr<span style="color: #000000;">&#91;</span>txtData.<span style="color: #0000FF;">dwStringCount</span><span style="color: #000000;">&#93;</span><span style="color: #008000;">;</span> <span style="color: #008080; font-style: italic;">// dest array</span>
Marshal.<span style="color: #0000FF;">Copy</span><span style="color: #000000;">&#40;</span>ptr, ptrs, <span style="color: #FF0000;">0</span>, ptrs.<span style="color: #0000FF;">Length</span><span style="color: #000000;">&#41;</span><span style="color: #008000;">;</span></pre></div></div>

<p>And finally we iterate through those pointers, marshalling the string pointed at by each:</p>

<div class="wp_syntax"><div class="code"><pre class="csharp" style="font-family:monospace;">var strings <span style="color: #008000;">=</span> <span style="color: #008000;">new</span> List<span style="color: #008000;">&lt;</span><span style="color: #FF0000;">string</span><span style="color: #008000;">&gt;</span><span style="color: #000000;">&#40;</span><span style="color: #000000;">&#41;</span><span style="color: #008000;">;</span>
<span style="color: #0600FF;">for</span> <span style="color: #000000;">&#40;</span>var i <span style="color: #008000;">=</span> <span style="color: #FF0000;">0</span><span style="color: #008000;">;</span> i <span style="color: #008000;">&lt;</span> ptrs.<span style="color: #0000FF;">Length</span><span style="color: #008000;">;</span> <span style="color: #008000;">++</span>i<span style="color: #000000;">&#41;</span>
<span style="color: #000000;">&#123;</span>
    strings.<span style="color: #0000FF;">Add</span><span style="color: #000000;">&#40;</span>Marshal.<span style="color: #0000FF;">PtrToStringAnsi</span><span style="color: #000000;">&#40;</span>ptrs<span style="color: #000000;">&#91;</span>i<span style="color: #000000;">&#93;</span><span style="color: #000000;">&#41;</span><span style="color: #000000;">&#41;</span><span style="color: #008000;">;</span>
<span style="color: #000000;">&#125;</span></pre></div></div>

<p>While the example I&#8217;ve presented here is specific to DnsQuery, the general approach should be applicable to any situation where you need to marshal a data structure containing a variable-length array.</p>
<p><a href="http://basildoncoder.com/marshal-variable-length-array.cs">Source code</a></p>
<p><a class="a2a_dd addtoany_share_save" href="http://www.addtoany.com/share_save"><img src="http://basildoncoder.com/blog/wp-content/plugins/add-to-any/share_save_171_16.png" width="171" height="16" alt="Share/Bookmark"/></a> </p>]]></content:encoded>
			<wfw:commentRss>http://basildoncoder.com/blog/2009/03/31/marshalling-a-variable-length-array-from-unmanaged-code-in-c/feed/</wfw:commentRss>
		<slash:comments>5</slash:comments>
		</item>
	</channel>
</rss>

<!-- Dynamic Page Served (once) in 0.380 seconds -->
