When the Problem is Not Them. It’s You.

Hello Dear Reader!  I’ve been the Data Platform Management Lead at Pragmatic Works for almost two years now.  It’s been an interesting journey that I’m enjoying.  I’m the tier 3 support in a lot of cases, play the role of manager, mentor, and fellow fighter in the trenches when required.  Today I wanted to write about something that deals with professional development.

In IT facts are facts, data is data, things perform well or they do not.  All of this leads to trust.  I do a lot of presentations on the Internals of SQL Server.  It is a fun and topic I enjoy it. *cough* nerd alert *cough*.  I do this not just for fun, but because knowledge is powerful.  It gives me the ability to let my clients have confidence that I understand what I’m talking about.  Very rare are any instances where I have been paid when I actually needed to flip hex to decimal and look through the binary that was produced.  The ability to display that knowledge when needed is crucial to ensuring you’ve got a second shot when you need it.

Projects, jobs, and opportunities come and go.  I’ve seen cases of spectacular success.  As anyone who has experienced success well knows, the path is littered with those who have failed before you.  Some projects are easy, some are hard.  All can be successful, but you have to have the right mindset.  When projects do fail it can be for a variety of reasons, typically it comes down to poor communication. 

I could be the typical marketing magazine and now list out ways projects fail, how communication is important, but Dear Reader you deserve better.  What you really want to know is how you fix it.  

IS IT TOO LATE FOR A RETRY

Unless all trust is gone it’s not too late.  When you are walking out the door, or being asked to walk out the door it is too late.  As long as you are still on the team, you should take that as validation to be part of the process.  In business everyone wants to be successful.  Everyone wants to win.  If you can help with that, then it is never too late.

The Upgrade Part 1

I was with a billion dollar company and we were upgrading their SQL Server boxes for their main application that the entire business ran off of from SQL Server 2000 to 2008 R2.  We had a very well thought out project plan, an itemized list and timeline of what needed to occur, and over 100 people including a team of VP’s that were on hand to monitor the event.  Leading up to it we hit a pretty big snag.

We were migrating the users from SQL Server 2000 to 2008 R2 and an odd thing happened.  The logins stopped working.  We caught it in Staging.  There was another consulting company that was also working with my client.  Often you will work with consultants or contractors for many different organizations, and you have solid and good relationships.  This was not one of those cases.  

The competing company took this opportunity to state very loudly that I didn’t know what I was doing.  That we could not just go from SQL 2000 to 2008 R2.  The hash had changed for logins, and you had to stutter step to SQL Server 2005, then re-export the logins to SQL 2008 R2.  This added an increased level of complexity, and threatened to blow our timeline.  I left the office dejected and the client questioning my experience.

You don't want to do IT.
You want to go home and rethink your life.

The EDW Part 1

I was with a client in the financial industry.  The mission to help pull data from a database, not SQL Server, to a new Enterprise Data Warehouse (EDW).  The plan would have us pull from our Online Transaction Processing (OLTP) system to move the data over.  They already had an Online Analytical Processing (OLAP) DW.  The data was Extracted, Transformed, and Loaded (ETL) from the OLTP system to the OLAP system.  The columns where not the same, they didn’t match up fully.  This was using a 3^rd party product so we didn’t have full visibility into the data mappings.

The business trusted the OLAP system.  Even though the OLTP was the source, they wanted us to validate our imported data against the OLAP, not the business rules and calculations provided to us using the raw OLTP data.

I didn’t agree with this approach.  A source system of this type that had pure 100% financial data, that financial decisions were made off of daily, should be the trusted source.  We should not be replicating the OLAP data but finding that the business calculations produced, validating them off of the source data and moving that data.  After many meetings on this subject I found myself in a room with the entire project team.  I realized that I was the only one passionately arguing this point of view.  In fact I found everyone was unified in one direction.  That direction was that I was wrong.

FIXING THE ISSUE (YOU)

Once you have realized that you are the issue the next step is to find a way to reestablish trust.  Learning, adapting, and putting the goals of team forward showing you can be a productive member goes a long way.

The Upgrade Part 2

When I started thinking about the issue I knew I wasn’t wrong.  Either that or I had gotten unbelievably lucky for every upgrade from 2000 to 200x that I’ve ever done.  Either way I wanted to know which it was.  If I had been lucky, well let's just say I would have a lot of phone calls I would need to make.

In order to be sure it was time to take a close look at the Microsoft provided scripts for transferring logins for SQL 2000 and guidance for SQL 2005 and up.  What I found was that the hash output was indeed different.  To test this I stood up a SQL 2000 box, made a user and a password and exported them.  I did the same on SQL 2005.  There were differences.  The length was different.  The 2000 scripts generated a password hash that was 102 characters long.  The 2005 and up were only 42 characters long.  Then it struck me those characters matched the SQL 2000 output exactally.

I used the SQL 2000 script to transfer the user to a SQL 2008 R2 instance.  Then I logged on as the user.  It worked.  I deleted the user, and ran the 2005 scripts.  It worked.  Those 42 characters were the important part.  That didn’t solve my issue with the client.  Our passwords were not working.  On a wild hare it struck me that the passwords were all lower case.  I tried camel casing them.  They worked.  I found by default that SQL 2000 passwords were case insensitive.

Poor password management had led them to record the passwords as lower case.  The hash had saved the camel case, and on export had enforced it by default on 2005 and up.  I brought this to my client the next day.  We realized that the passwords they had stored were not always correct.  In order to fix this we had to reset and update every application and it’s password before we could proceed. 

We did, and the upgrade rolled on.

The EDW Part 2

I stood at the front of the room and realized every eye was on me.  I realized that the business firmly believed in the process they had laid out.  I also realized, I was the only one not on board.  This was a definitive moment.  I could stand my ground, but very quickly I could see that would lead me off this project.  Failure is not an option.  So I went for option 3.

The process of not trusting the source is, and this is an understatement, not good.  Even with that staring me down, I could see an opportunity.  If I was onboard with the plan, I could work from the inside to validate that the source data was what we needed.

So I did just that.  I wasn’t excited, but I smiled big.  I wasn’t 100% convinced, but I threw in my hat, hard work, and rolled up my sleeves.  That first day the grin on my face may have been strained, by the end of the next 4 weeks it wouldn’t be at all.

We didn’t have all the business rules or the ETL logic for the 3^rd party data warehouse that the business trusted.  The people in the room knew that.  They were committed to getting this right, and putting the puzzle pieces together.  The process would be painstaking, but as I made my change of heart and discussed the validation we would need to do, mapping documents we would need to create, business logic we would need to confirm, validate, and how we would validate the imports it started to come together.

PULLING IT ALL TOGETHER

“So Balls,” you say, “That’s great that you fixed it. How do we fix it?”

Great question Dear Reader.  These stories both had a positive outcome.  They happened because I didn’t give up, and I continued to work to help the team succeed.  I don’t want to paint this as if I’m never wrong.  Just ask Adam Jorgensen (@AJBigdata | Blog), he’ll tell you that I’m wrong quite often.  I’ve also blogged about learning from mistakes, mistakes are important.  Be humble in the way you interact and listen to people.  People who believe they are never wrong, suffer defeat far worse than those of us that recognize it as part of growing.  

 

If I was wrong about the passwords from 2000 to 200x I wanted to know.  I wanted to understand why, and what we could do to fix it.  Had I been wrong, I would have sung it from the mountain tops.  I would have blogged on it and gladly advised people on it.  Instead I found the answer I was looking for and the logic to back it up, I share that just as freely as I would have my failure.

When everyone is fully committed to an idea that you do not agree with instead of assuming they are wrong, consider that you are.  It changed my point of view on the project.  It allowed me to embrace a team that wanted someone to take them on the path towards success and be part of that team.  It also reinforced a very simple fact, I am not always right and I need to listen to others.

In each case you have to want to achieve the end goal, and that has to be more important than being right.

As always Dear Reader, thanks for stopping by!

Thanks,

Brad