Monday, July 9, 2012

SQL Internals Reading Data Records Part 6: Variable Length Data




Today we will be discussing what that array was pointing to the Variable Length Data.  The nice part is this should be easy going for you by now because this is very similar to what we did in Part 3.  We are reading data. 


One of the things that we will use today is what we learned last time to bring it all home.  So without fan fair or ado let’s dive right into it.


VARIABLE LENGTH DATA
First let’s update our chart so we know what part of the Data Record we are tackling.  Once again these images come by way of Paul Randal (@PaulRandal | Blog) and the MCM Video series on Data Structures and the good people from Microsoft.


The variable length columns are the last bit of data that we need to read out of a regular data record.  The offset array that we looked at yesterday gives us a value for the end of the variable length record we are looking for so that we can find the end of our data record and read it in reverse.

We’ll use the same code we’ve been using for the last couple days but in case you are missing it here it is.  First we’ll create our table and insert a record.
IF EXISTS(SELECT NAME FROM sys.tables WHERE NAME='dataRecord2')
BEGIN
       DROP TABLE dataRecord2
END
GO
CREATE TABLE dataRecord2
                     (myID INT
                     ,myfixedData CHAR(4)
                     ,myVarData1 VARCHAR(6)
                     ,myVarData2 VARCHAR(9)
                     ,myVarData3 VARCHAR(6)
                     )

GO
INSERT INTO dataRecord2(myID, myfixedData, myVarData1, myVarData2, myVarData3)
VALUES (7, 'XXXX', 'SSS', 'WWWWWWWWW', 'BB')
GO

Now let’s do a DBCC IND and get our page numbers.

DBCC IND(demoInternals, 'dataRecord2',  1)
GO

Remember that a page type 10 in an allocation page and we are reading a data page.  So look for the page number that has a PageType=1.  We’ll follow that up with a DBCC PAGE on Page 296, remembering to turn on Trace Flag 3604 so that we get our output to our SSMS window.  *Remember your page numbers may be different than mine. 

DBCC TRACEON(3604)
GO
DBCC PAGE('demoInternals', 1, 296, 1)
GO

Now that we’ve got our page let’s take a look at it, I’m only going to post the relevant output.
Record Type = PRIMARY_RECORD        Record Attributes =  NULL_BITMAP VARIABLE_COLUMNS
Record Size = 37                   
Memory Dump @0x0000000014DEA060

0000000000000000:   30000c00 07000000 58585858 05000003 001a0023  0.......XXXX.......#
0000000000000014:   00250053 53535757 57575757 57575742 42        .%.SSSWWWWWWWWWBB

The data that we are looking at is in bold and underlined and highlighted in red.  If we use the variable offset array values we could find which data belongs to which column.  So let’s do that real quick.  Don’t forget to use our handy Hex to decimal and binary conversion tool, http://easycalculation.com/hex-converter.php.


Each block of data in a record is 8 characters long but stands for 4 bytes of information.  So if we look at all of the blocks we see we have 9 * 4 is 36 bytes.  The last bit of information is 42 is two characters long and is 1 byte long.  So our total record is 37 bytes long.  Since variable length data can have different lengths that changes with the values in place we need to look at the variable length array to find our data and read it forward.

Our first Variable length column is 53 5353. Because this is regular character data we will translate one byte at a time.  0x53 translates to binary 01010011, if we use our binary to ascii conversion tool, , then we find that binary value translates to S.  There are three 53 values and our first variable length column has ‘SSS’ as it’s the value that we inserted into myVarData1.  Following this pattern you could very easily translate the contents of the other two variable length columns.

I’ll leave you to it Dear Reader, as always thanks for stopping by.

Thanks,

Brad

Thursday, July 5, 2012

Differences in the Null Bitmap between SQL 2005 and 2012


Hello Dear Reader, I’ve been working on a series for BIDN.com, the Business Intelligence Developers Network, on how to read a Data Record.  I’ve always wanted to take the time to go rummaging around in the internals of SQL Data Records and this was my shot to do it.  If you are interested please feel free to read Part 1 Tag bytes, Part 2 The Null Bitmap Offset, Part 3 Fixed Data, and Part 4 The Null Bitmap.

While I was working on part 4 I found that the results that I was getting for the Null Bitmap were a bit different than what I was finding on the blogosphere and the Intrawebs.  I was reading the blogs of some very smart people, so that lead me to one conclusion: I was doing it wrong.  So I started following their examples and comparing the results and well….there’s only so may ways you can type DBCC PAGE before you start thinking maybe I’ve found something.  I was working in SQL 2012 so my first thought was that it changed there.  So I stood up a VM with SQL 2008 R2 on it and I got the same results.  So then I stood up another VM with 2005 and Viola! I was looking at results just like everyone else.  So then I stood up ANOTHER VM with 2008 RTM and once again I was back to what I was initially looking at.

“So Balls,” you say, “What was different?”

Sorry for the Ramble Dear Reader, what I found was that in 2005 when you translate the Null Bitmap if you had a number of columns that doesn’t divide easily into 8, (The Null Bitmap is one byte which translates out to a map of 8 individual bits, and you get 8 whether you need them or not), instead of using 1’s for unallocated bits they use 0’s. 

I understand not exactly a jaw dropping conclusion but if you’re a geek like me that likes to dive into internals this should be a fun read.

LET’S ROCK

So now that we’ve got some extra detail out of the way let’s dive into examples.  First we’ll create our table with two fixed length columns and three variable length.  We’ll keep the variable length columns as nulls at first so we can see how the Null Bitmap changes.

IF EXISTS(SELECT NAME FROM sys.tables WHERE NAME='dataRecord2')
BEGIN
       DROP TABLE dataRecord2
END
GO
CREATE TABLE dataRecord2
                     (myID INT
                     ,myfixedData CHAR(4)
                     ,myVarData1 VARCHAR(6)
                     ,myVarData2 VARCHAR(6)
                     ,myVarData3 VARCHAR(6)
                     )

GO
INSERT INTO dataRecord2(myID, myfixedData, myVarData1, myVarData2, myVarData3)
VALUES (7, 'XXXX', null, null, null)
GO

Now that we’ve got our table and a record inserted let’s use DBCC IND to find our page.

DBCC IND(demoInternals, 'dataRecord2',  1)
GO


Remember type 10 is an allocation page and we want a data page or a PageType 1, we see that Page 288 is our candidate.  Now let’s use DBCC Page to get page 288, don’t forget to turn on Trace Flag 3604.  I’m only going to copy the relevant bits from our page dump and remember your page numbers may differ.  I’m using SQL 2012 for this output.

DBCC TRACEON(3604)
GO
DBCC PAGE('demoInternals', 1, 288, 1)
GO

Record Type = PRIMARY_RECORD        Record Attributes =  NULL_BITMAP    Record Size = 15

Memory Dump @0x00000000136EA060

0000000000000000:   10000c00 07000000 58585858 05001c             ........XXXX...

Here in Red, Green, and Bold (sorry to anyone who is color blind hence the bold…), is our portion of the record slated as our Null Bitmap.  0500 is the first two bytes of our record, remember two numbers equal a Hex pair.  Our Pairs are 05 & 00, they are flipped so you would read them as Hex 0x0005, which translates out to 5 and we do have 5 columns.  Don’t forget to use our handy Hex to binary convertor, http://easycalculation.com/hex-converter.php. The last portion of the record is the Null Bitmap itself 1c.  In Hex this works out as 0X1c and converts to binary as 00011100.  In order to read this binary value accurately we flip it and it becomes 0011100



We break this down per column, because we have two columns that are not null three Variable length columns that are null and then the remaining two 0’s are not in use.  So why are they are you may ask?   Because 1 byte equals 8 bits, and these are left over bits.

Now this is one of the differences that I’ve noticed from 2005 to 2008 on up, in 2005 the left over bits where marked as 1’s.  Let’s take a look real quick to back that up.  I set up a VM running Windows 2008 R2 and SQL 2005 Straight up RTM no Service Packs applied.  We’ll run the exact same scripts as above and look at the Null Bitmap output.


IF EXISTS(SELECT NAME FROM sys.tables WHERE NAME='dataRecord2')
BEGIN
       DROP TABLE dataRecord2
END
GO
CREATE TABLE dataRecord2
                     (myID INT
                     ,myfixedData CHAR(4)
                     ,myVarData1 VARCHAR(6)
                     ,myVarData2 VARCHAR(6)
                     ,myVarData3 VARCHAR(6)
                     )

GO
INSERT INTO dataRecord2(myID, myfixedData, myVarData1, myVarData2, myVarData3)
VALUES (7, 'XXXX', null, null, null)
GO

Now that we’ve got our table and a record inserted let’s use DBCC IND to find our page. Notice we use DBCC IND with a -1 if you attempt this with the exact same script as above you’ll get an output error for dump style 1.

DBCC IND(demoInternals, 'dataRecord2',  -1)
GO


Remember type 10 is an allocation page and we want a data page or a PageType 1, we see that Page 153 is our candidate.  Now let’s use DBCC Page to get page 153, don’t forget to turn on Trace Flag 3604.  I’m only going to copy the relevant bits from our page dump and remember your page numbers may differ. 

Record Type = PRIMARY_RECORD         Record Attributes =  NULL_BITMAP    
Memory Dump @0x0000000008BAC060

0000000000000000:   10000c00 07000000 58585858 0500fc††††........XXXX... 
asdf
As you can see from above we get the output of fc in SQL 2005 instead of 1c like we got in SQL 2012.  If you take the Hex for 0xfc and dump it to binary you get 11111100.  When when we flip works out to 00111111.



Just to reiterate, unless you want to validate this for yourself, I stood up VM’s for 2005 RTM, 2008 RTM, 2008 R2 RTM, and SQL 2012 RTM.  Only 2005 handles unused bit’s as 1’s the rest treat them as 0’s.  The Null Bitmap was an optimization that was added in 2005, Read Paul Randal’s, Paul(@PaulRandal | Blog), blog post on that here.

Since it was introduced in 2005 then there I’m sure there was a good reason to change it in 2008 and up.  My guess is that instead of masking as columns that where just there of offset the bitmap some logic is used based on the number of columns and the bitmap to determine how much of the record remains to be read.

Whatever the case a lot of the documentation I came across regarding this all pointed to the unused bit’s being labeled as 1’s.  That through me for a loop at first and I just wanted to toss this out there to anyone else searching on the subject.

Thanks for stopping by,

Brad