Let's Talk About Indexes: An Introduction

A very good friend of mine, whose got this kick-ass job working for the President J, wrote me a question about indexes.  And I thought how better than a phone call and a follow up blog talking about indexes in general.  The question centered on the uses of Clustered Indexes vs. Non-Clustered Indexes and the use for each.

CLUSTERED vs NON-CLUSTERED

The good news is this really isn't a VS kind of thing.  Clustered Indexes and Non-Clustered Indexes are both indexes.  They are both made of of an internal B-Tree structure.  

When you think of a B-Tree, think of your standard Hierarchy.  There are 3 named levels to a B-Tree structure.  The Root Level which is our top level and contains values that direct a path towards the next level, the Intermediate level and just like the Root Page points to the next level of Data, and finally the Leaf Page.

One of difference between a Clustered Index and a Non-Clustered Index is that a Clustered Index physically sorts the Data, based off of the Clustered Index.  Another difference is you can only have one Clustered Index per Table, but you can have multiple Non-Clustered Indexes.  The Clustered Index is most commonly thought of as a Primary Key, (the Clustered Index doesn't have to be the Primary Key, but that could get a little confusing so we'll save that for another day).

So we'll say that our B-Tree up above is using the letters of the Alphabet as it's Clustered Index.  I'll issue a quick T-SQL Select to get our letter.

SELECT

    *

FROM dbo.alphabet

WHERE

    letter='G'

If I wanted to go get the letter G, at the Root Level I deduce that G comes after A and before J so I would take a pointer and go to the Intermediate level that contained A, C, F, and H, From there I would further deduce that G is after F and before H, and I would travel down to our Leaf Page where I would find the letter G.

*IF your query's first Search Argument is the Clustered Index, and your range of values are narrow, then you will use your Index structure in a manner called a Seek.  If you have to read the entire table to get the contents of your information back, it is called Scan.  Scan's have their place and are not necessarily a bad thing.  But it is entirely dependent on the amount of data being returned, and the route taken to search for it.

One of the best examples I can think of to describe what an Index does, and further illuminate our B-Tree Structure, and show how  Clustered Indexes and  Non-Clustered Indexes  work together is the good old Phone Book.  If I told you to find Bradley Ball's address in the phone book you've got 2 ways of finding my name.

1. Turn through the book 1 page at a time until you get to Bradley Ball, and then read his address.

2. Flip to the B's (our Clustered Index) and scan until you get to Ball.  Now Scan Ball for the first name of Bradley (our Non-Clustered Index), and find the address.  You just performed a Seek, using the search arguments of Last Name and First Name to find Address.  

So now let's toss out a little vocabulary, to help further this discussion along.

HEAP- A heap is a table that has no clustered index.  Data is stored on a head in the order that it is inserted.  This means it is easily fragmented, and could require scanning as selecting data would require a full scan of the heap in order to find it.

CLUSTERED INDEX- A Clustered Index is a Unique value, that determines the physical sort order of the data as it is stored on a page.  The leaf level of a Clustered Index is the data of the table.  Clustered indexes have a limited key size of 900 bytes.

NON-CLUSTERED INDEX- A Non-Clustered Index is an index placed on a row in a Table, the leaf level of a Non-Clustered Index contains the data that makes up the Non-Clustered Index, as well as a physical pointer to the leaf level of the Clustered Index.  You can have many Non-Clustered Indexes on a Clustered Index Table, or a Heap.  Non-Clustered Indexes have the same 900 byte limitation and are limited to 16 columns, you can get around this using Include Columns.

"So Balls", you say, "Indexes sound great, we should put them on EVERYTHING! Right?"

I'm Glad you asked Dear Reader, and the response, regarding OLTP systems, is NO.

BALANCING ACT

Every Index that you place on a table has a cost of additional overhead.  You want to use them to speed up your database, but if you're not careful you could grind it to a halt.  

If you have a table with a Clustered Index, when you insert and update pages, and GROW the data, you are increasing your over head, slightly and it's worth it, to make sure that your Root and Intermediate level pages have the correct values to point to your leaf pages.

When you add a Non-Clustered Index, every time you insert a value for that Index you write to your table AND your Non-Clustered Index.  So if you Add another, and another, and another, then suddenly every time you write 1 record to your table, you could have many, many, many writes to keep all of your Indexes updated.

One of the first things I look at when I get a call about a "performance" problem on a database is to look at the indexes.  I've had instances where I have found tables with more indexes than columns.

COVERING INDEXES

The final high level topic that I want to talk about is Covering Indexes.  A Covering Index is an index that satisfies all of the return requirements of a query.  You could take all of the fields that you want to return, limit 16, in a query and if you add them to a single Non-Clustered Index you can do so.

One way to make a covering index is to add each column to the Non-Clustered index, another is to add additional columns as Include Columns.  Include columns allow you to get around the 900 byte limitation of an index, because Include columns are not stored as key values.  There are some data types that are not allowed to participate in Included Columns, Text and Image datatypes.  You can have 1023 Included columns on an index.

I just want to point out thought that, 1023 should not be a goal.  Every value you add, will add additional overhead to your leaf level Non-Clustered index.  Be sure to take baselines before and after, in order to know if your Index changes are having a positive affect on your query performance.




Thanks,

Brad

Don't Be Afraid, Not Being The Smartest Person In The Room

"I don't have to be the smartest person in the room". I once didn't take a job because of this phrase.  It was probably one of the best things that ever happened to me and it worked out quite well for a friend of mine.  But I'm getting ahead of myself, let me tell the story from the beginning.

DON'T BE AFRAID

A little philosophy first.  I'm not going to go Ayn Rand or Immanuel Kant on you, but as Louis Davidson (Blog|Twitter) would say, "We're drinking my flavor of Kool-AID today.

I don't have to be the smartest person in the room.  I don't, just don't need to.  Sometimes you will be, sometimes you won't.  If you don't worry about it, then it takes a lot of pressure and stress off.

"But Balls", you say, "Why are you afraid of being smart!?"

Ahh but that's just it Dear Reader, I'm not Afraid.

We've all been in the room with someone that THINKS they know everything.  We've all worked with someone, that just always had to be right.  Have you ever worked with someone like this on the day they were wrong?  

I have a better question would you want to be them on the day they are proven wrong?  No I wouldn't either.  If you always HAVE to be right, then being wrong is a crushing blow.

People can be afraid of a lot of things.  They can let themselves be ruled by fear, fear that your boss won't listen, fear that you don't know as much as the guy sitting next to you, Fear of Not Being the Smartest Person in the room.

I've seen this back fire on so many people that I've always wanted to take a different approach.  I find that you never know where the best idea will come from on a team.  It could be the developer that has been written off as lazy, that has a great out of the box idea.  It could be the obnoxious DBA that has some insight into the best way to layout a project's architecture.  It could be a Server Engineer that has really good insight on code management and migration.

If you make yourself open to the possibility that everyone has something to contribute, then you don't close yourself off to ideas.  And again you take a lot of weight off of your shoulders.

ROOM TO GROW AND STRETCH

A couple jobs ago I was looking for a new position.  I had reached what I felt was a ceiling.  I had been a Sr. Developer, mentored others, gotten my SQL Certifications, and filled the roles of a Sr. DBA & Sr. Developer.  I had worked on some cool projects, and really loved the people I had worked with.  

However, I wanted to grow and we were a very top heavy team.  A lot of people had been with the company for years, many had their Masters in IT or MBA's, MCAD's, and MCSE's brilliant people that I'm still very happy to call friends.  But with a great experienced group the only way to move into the full time job I wanted was for someone to leave.

HE'S NOT THE MIZ BUT HE'S AWESOME!

So I struck out to find greener pastures.  One particular job had advertised itself as Sr. DBA position.  I applied, and found that the person leaving the job was Awesome.  The company hated to loose Awesome, the client that Awesome served was thrilled with them, and Awesome had no desire to go but was moving to follow his wife's career.

We met several times over a month, and I went on-sight to meet the client.  The more I learned about the position, the more it didn't feel like a good fit.  I found that the only reason they wanted a DBA and were requiring an MCITP for SQL was because Awesome had that and they wanted Awesome II.  It was more of an Analyst/Team Lead position.

While meeting the client, I was asked to explain how I would describe myself as a Manger.  And I said "I'm not afraid, not to be the smartest person in the room.  I think everyone brings something to the table, I like to listen, I like to consider what the boots on the ground have to say, I don't think that my position entitles me to ideas that are more brilliant than anyone else, and as a Manager I'm not afraid to cheer someone on who's smarter than I am in a subject.  If I find someone smarter than I am, I want to learn from them, so I can be as educated as possible on the subject at hand."

I'M THE BEST AT WHAT I DO EVEN THOUGH WHAT I DO ISN'T VERY NICE IS BE A DBA

To be honest I thought it was a pretty good answer, and I'm still satisfied with it.  When we left the client site, the person who would have been my future manager, tore into me. 

I believe the coversation started out with "I DON'T KNOW WHAT IN THE HELL THAT SHUCK'S GOOD OLD BOY, HUMBLE CRAP WAS......" Needless to say I was taken back.   But I stuck to my guns, which probably only served to enrich the colorful nature of the rest of our conversation.  

The remainder of the conversation centered on how I always, ALWAYS, needed to be the smartest person in the room.  As I explained a little earlier, this flies in the face of how I work.  You can take a job where you compromise yourself, but I can't see how you can do that and be happy.

I told my wife later, that had I already taken the job the tone of the conversation would have left me worried that I would have been fired.  We talked about it that evening, and even though it would have been a significant raise, SIGNIFICANT RAISE, I asked if she would be alright with me passing.  My very wise and beautiful wife replied by saying that she didn't know why I was still even considering it.

IT ALL WORKED OUT

The day before I decided to pass a brilliant friend of mine, Perfect, found themselves without a job.  When I had worked with Perfect previously, Perfect had been an Analyst that had been a Team Lead over Developers, and was a perfect for this job.

I got Perfect's resume, and called and informed the recruiter I had been dealing with that I wouldn't be taking the job replacing Awesome.  Despite protests, I told them it may not seem like it, but I've got a person for this job whose Perfect.  I didn't have any hard feelings.  I wasn't the right person, and they weren't the right fit for me.  But I was going to send them Perfect's resume, and they should set up a meeting as soon as possible.

Perfect still works there, Perfect loves their job and is very successful.  And me, I found a little job up in D.C. working for the President, that led me to the Great State of Florida where I was very happy to land.

I guess my only parting advice Dear Reader would be don't be afraid of a someone who is smarter than you.  Embrace them, learn from them, and work hard.  At the end of the day how can anybody be unhappy with learning something new, and making themselves better.

Thanks,

Brad

  

T-SQL Tuesday 17: Apply

Welcome to another T-SQL Tuesday!  This week is hosted by Matt Velic (blog|twitter), so Matt what is this week about?

APPLY Knowledge

Recently on Twitter, I heard the claim that “If you don’t understand the APPLY operator, your skills are somewhere around the 50th percentile at best.” While I believe that Adam was giving a warning to self-proclaimed experts (possibly one he might have been interviewing at the time…), I also believe that we could take it as a challenge as a T-SQL blogging community to learn more about APPLY and the ways in which we can use it in our work.

Please share how you use this wonderful feature. Maybe you know how APPLY works inside and out? Perhaps you’ve got a fantastic user defined function (UDF) to share? Or maybe your experience revolves around using Dynamic Management Functions (DMFs) in your never-ending quest for SQL Server performance? Let the community know as it is time to study!

All right, here we go then this is all about the Apply!

START WITH THE BASICS

The Apply operator was added to SQL Server with SQL 2005.  Apply  is a logical join operator, it allows you to perform a join on a Table Valued Function.  I think the most common way that people use Apply is to get the SQL Text or Query Plan using DMV’s.

SELECT

    er.session_id

    ,st.text

    ,qp.query_plan

FROM sys.dm_exec_requests er

CROSS APPLY sys.dm_exec_sql_text(er.plan_handle) st

CROSS APPLY sys.dm_exec_query_plan(er.plan_handle) qp

The previous query will return any currently executing query on your system.  Using the CROSS APPLY is like using a left join, but on a Table Value Function, there is also an OUTER APPLY which is similar to the functionality as a right join.  In this case we are Using CROSS APPLY to return the Text of the SQL Statement being executed, as well as the Query Plan.

The Query Plan can have some over head associated with it, if you’re on a busy system I would recommend commenting out the Query Plan unless you’re truly looking for it.

SURE BUT CAN IT PLAY THE VIOLIN

So the DMV route is pretty well known, but how do you apply this in a non-troubleshooting capacity?  For this next example I’m using the AdventrueWorks2008R2 Database.

So let’s toss out a Case Scenario.  Our boss has come to us, there is an application that the company uses and as part of the expanded functionality we’ll be adding a lookup of Customer’s information.   We need to provide a list of Customer Names on demand, or the Name of a particular Customer, that are identified to us as People and not businesses.  (Not a complicated Scenario, but hey let’s start out practical).

If you Expand AdventureWorks2008R2 | Programmability | Functions| Table-valued Functions, you’ll see dbo.ufnGetContactInformation.  This Table Valued Function is exactly the type that we can use in the CROSS APPLY Statement.

Our Table Value Function accepts the @PersionID parameter, and we will be using the SALES.CUSTOMER table.   We take a quick look at the table to see what details we pull back.

SELECT

    *

FROM

    SALES.Customer

We see that names are not stored in this table, but we know we can retrieve them through our function.  Okay we don’t know that, but if you right click on the function and script it out you can read through it and see what it is doing.

So let’s use our CROSS APPLY statement.

SELECT

    ci.FirstName

    ,ci.LastName

    ,ci.JobTitle

    ,sc.AccountNumber

   

FROM

    SALES.Customer sc

    CROSS APPLY dbo.ufnGetContactInformation(sc.PersonID) ci

So now we can see that this can be used in a non-troubleshooting capacity.

 So let’s think about a couple other possibilities.  Let’s select an order being delivered and we want to know the last update in its location.  We have a customer looking at an item on a web page, and we want to display a coupon for the item.

There are a lot of possibilities for CROSS APPLY, I would suggest watching  the MCM Waits and Queues video by Paul Randal (blog|twitter) this link to the Demo will cover a great scripts by Glenn Berry (blog|twitter) using CTE’s & one by Joe Sack (blog|twitter) that makes a really good use of OUTER APPLY.

/*

How do you clear the server's wait

stats?

DBCC SQLPERF('sys.dm_os_wait_stats', CLEAR);

=================================================

BY: Glenn Berry & Paul Randal

CTE to get wait stats.

This CTE get's the wait stats as they are occuring on the

system and give's you the total seconds of wait, the

# of occurances, as well as the top percentage of where

your wait time is coming from. 

The Exclusions of certian wait types were recommendations

by Paul Randal in the MCM wait types video series

==================================================

*/

WITH Waits AS

(SELECT

    wait_type,

    wait_time_ms /1000.0 AS Waits,

    (wait_time_ms - signal_wait_time_ms)/1000.0 AS Resources,

    signal_wait_time_ms/1000.0 AS Signals,

    waiting_tasks_count AS WaitCount,

    100.0 * wait_time_ms /SUM(wait_time_ms) OVER()AS Percentage,

    ROW_NUMBER()OVER

    (ORDER BY wait_time_ms DESC)AS RowNum

FROM sys.dm_os_wait_stats

WHERE

    wait_type NOT IN ('CLR_SEMAPHORE','LAZYWRITER_SLEEP','RESOURCE_QUEUE','SLEEP_SYSTEMTASK','SQLTRACE_BUFFER_FLUSH','WAITFOR','LOGMGR_QUEUE','CHECKPOINT_QUEUE'

,'REQUEST_FOR_DEADLOCK_SEARCH','XE_TIMER_EVENT','BROKER_TO_FLUSH','BROKER_TASK_STOP','CLR_MANUAL_EVENT'

,'CLR_AUTO_EVENT','DISPATCHER_QUEUE_SEMAPHORE','FT_IFTS_SCHEDULER_IDLE_WAIT'

,'XE_DISPATCHER_WAIT','XE_DISPATCHER_JOIN','SQLTRACE_INCREMENTAL_FLUSH_SLEEP')

)

SELECT

    W1.wait_type,

    CAST(W1.Waits AS DECIMAL(12,2))AS Wait_S,

    CAST(w1.Resources AS DECIMAL(12,2))AS  Resources_S,

    CAST(W1.Signals AS DECIMAL(12,2))AS Signal_S,

    W1.WaitCount,

    CAST(W1.Percentage AS DECIMAL(12,2))AS Percentage,

    CAST(AVG(W1.Signals)/ W1.WaitCount AS DECIMAL(12,8))AS AvgWait_S,

    CAST(AVG(W1.Resources)/ W1.WaitCount AS DECIMAL(12,8))AS AvgRes_S

FROM

    Waits AS W1

    INNER JOIN Waits AS W2

ON

    W2.RowNum <=W1.RowNum

GROUP BY

    W1.RowNum,W1.wait_type,W1.Waits,W1.Percentage, w1.Resources, w1.WaitCount,w1.Signals

HAVING

    SUM(W2.Percentage) -W1.Percentage <99

/*

=================================================

BY: Joe Sack

This will give you the waits as they are happening

on a system, and give you the query text, as well

as the execution plans as they are occuring

I added the @@SPID to the query so we would not

see our own query plan come back while trouble

shooting

==================================================

*/

Select

    DB_NAME(est.dbid) AS DatabaseName,

    owt.session_id,

    owt.wait_duration_ms,

    owt.wait_type,

    owt.blocking_session_id,

    owt.resource_description,

    es.program_name,

    est.text,

    est.dbid,

    eqp.query_plan,

    es.cpu_time,

    es.memory_usage

from sys.dm_os_waiting_tasks owt

INNER JOIN sys.dm_exec_sessions es ON

    owt.session_id=es.session_id

INNER JOIN sys.dm_exec_requests er ON

    es.session_id=er.session_id

OUTER APPLY sys.dm_exec_sql_text(er.plan_handle) est

OUTER APPLY sys.dm_exec_query_plan(er.plan_handle) eqp

WHERE es.is_user_process=1

    AND es.session_id<> @@spid;

 

Here are the Scripts, but go watch to video to see how to use them!

Thanks,

Brad