Hello Dear Reader!
I’ve been the Data Platform Management Lead at Pragmatic Works for
almost two years now. It’s been an
interesting journey that I’m enjoying.
I’m the tier 3 support in a lot of cases, play the role of manager,
mentor, and fellow fighter in the trenches when required. Today I wanted to write about something that
deals with professional development.
In IT facts are facts, data is data, things perform well or
they do not. All of this leads to
trust. I do a lot of presentations on
the Internals of SQL Server. It is a fun
and topic I enjoy it. *cough* nerd alert *cough*. I do this not just for fun, but because
knowledge is powerful. It gives me the
ability to let my clients have confidence that I understand what I’m talking
about. Very rare are any instances where
I have been paid when I actually needed to flip hex to decimal and look through
the binary that was produced. The
ability to display that knowledge when needed is crucial to ensuring you’ve got
a second shot when you need it.
Projects, jobs, and opportunities come and go. I’ve seen cases of spectacular success. As anyone who has experienced success well
knows, the path is littered with those who have failed before you. Some projects are easy, some are hard. All can be successful, but you have to have
the right mindset. When projects do fail
it can be for a variety of reasons, typically it comes down to poor
communication.
I could be the typical marketing magazine and now list out
ways projects fail, how communication is important, but Dear Reader you deserve
better. What you really want to know is
how you fix it.
IS IT TOO LATE FOR
A RETRY
Unless all trust is gone it’s not too late. When you are walking out the door, or being
asked to walk out the door it is too late.
As long as you are still on the team, you should take that as validation
to be part of the process. In business
everyone wants to be successful.
Everyone wants to win. If you can
help with that, then it is never too late.
The Upgrade Part 1
I was with a billion dollar company and we were upgrading
their SQL Server boxes for their main application that the entire business ran
off of from SQL Server 2000 to 2008 R2. We
had a very well thought out project plan, an itemized list and timeline of what
needed to occur, and over 100 people including a team of VP’s that were on hand
to monitor the event. Leading up to it
we hit a pretty big snag.
We were migrating the users from SQL Server 2000 to 2008 R2
and an odd thing happened. The logins
stopped working. We caught it in
Staging. There was another consulting
company that was also working with my client.
Often you will work with consultants or contractors for many different
organizations, and you have solid and good relationships. This was not one of those cases.
The competing company took this opportunity
to state very loudly that I didn’t know what I was doing. That we could not just go from SQL 2000 to
2008 R2. The hash had changed for
logins, and you had to stutter step to SQL Server 2005, then re-export the
logins to SQL 2008 R2. This added an
increased level of complexity, and threatened to blow our timeline. I left the office dejected and the client
questioning my experience.
|
You don't want to do IT. You want to go home and rethink your life. |
The EDW Part 1
I was with a client in the financial industry. The mission to help pull data from a database,
not SQL Server, to a new Enterprise Data Warehouse (EDW). The plan would have us pull from our Online
Transaction Processing (OLTP) system to move the data over. They already had an Online Analytical Processing
(OLAP) DW. The data was Extracted,
Transformed, and Loaded (ETL) from the OLTP system to the OLAP system. The columns where not the same, they didn’t
match up fully. This was using a 3rd
party product so we didn’t have full visibility into the data mappings.
The business trusted the OLAP system. Even though the OLTP was the source, they
wanted us to validate our imported data against the OLAP, not the business
rules and calculations provided to us using the raw OLTP data.
I didn’t agree with this approach. A source system of this type that had pure
100% financial data, that financial decisions were made off of daily, should be
the trusted source. We should not be
replicating the OLAP data but finding that the business calculations produced,
validating them off of the source data and moving that data. After many meetings on this subject I found
myself in a room with the entire project team.
I realized that I was the only one passionately arguing this point of
view. In fact I found everyone was
unified in one direction. That direction
was that I was wrong.
FIXING THE ISSUE
(YOU)
Once you have realized that you are the issue the next step
is to find a way to reestablish trust.
Learning, adapting, and putting the goals of team forward showing you
can be a productive member goes a long way.
The Upgrade Part 2
When I started thinking about the issue I knew I wasn’t
wrong. Either that or I had gotten unbelievably
lucky for every upgrade from 2000 to 200x that I’ve ever done. Either way I wanted to know which it was. If I had been lucky, well let's just say I would have a lot of phone calls I would need to make.
In order to be sure it was time to take a
close look at the Microsoft provided scripts for
transferring logins for SQL 2000
and
guidance for SQL 2005 and
up. What I found was that the hash
output was indeed different. To test
this I stood up a SQL 2000 box, made a user and a password and exported
them. I did the same on SQL 2005. There were differences. The length was different. The 2000 scripts generated a password hash
that was 102 characters long. The 2005
and up were only 42 characters long.
Then it struck me those characters matched the SQL 2000 output
exactally.
I used the SQL 2000 script to transfer the user to a SQL
2008 R2 instance. Then I logged on as
the user. It worked. I deleted the user, and ran the 2005
scripts. It worked. Those 42 characters were the important part. That didn’t solve my issue with the
client. Our passwords were not
working. On a wild hare it struck me
that the passwords were all lower case.
I tried camel casing them. They
worked. I found by default that SQL 2000
passwords were case insensitive.
Poor password management had led them to record the
passwords as lower case. The hash had
saved the camel case, and on export had enforced it by default on 2005 and
up. I brought this to my client the next
day. We realized that the passwords they
had stored were not always correct. In
order to fix this we had to reset and update every application and it’s password
before we could proceed.
We did, and the upgrade rolled on.
The EDW Part 2
I stood at the front of the room and realized every eye was
on me. I realized that the business
firmly believed in the process they had laid out. I also realized, I was the only one not on
board. This was a definitive
moment. I could stand my ground, but
very quickly I could see that would lead me off this project. Failure is not an option. So I went for option 3.
The process of not trusting the source is, and this is an
understatement, not good. Even with that
staring me down, I could see an opportunity.
If I was onboard with the plan, I could work from the inside to validate
that the source data was what we needed.
So I did just that. I
wasn’t excited, but I smiled big. I wasn’t
100% convinced, but I threw in my hat, hard work, and rolled up my
sleeves. That first day the grin on my
face may have been strained, by the end of the next 4 weeks it wouldn’t be at
all.
We didn’t have all the business rules or the ETL logic for
the 3rd party data warehouse that the business trusted. The people in the room knew that. They were committed to getting this right,
and putting the puzzle pieces together.
The process would be painstaking, but as I made my change of heart and
discussed the validation we would need to do, mapping documents we would need
to create, business logic we would need to confirm, validate, and how we would
validate the imports it started to come together.
PULLING IT ALL
TOGETHER
“So Balls,” you say, “That’s great that you fixed it. How do
we fix it?”
Great question Dear Reader.
These stories both had a positive outcome. They happened because I didn’t give up, and I
continued to work to help the team succeed.
I don’t want to paint this as if I’m never wrong. Just ask Adam Jorgensen (
@AJBigdata |
Blog), he’ll tell you that I’m wrong quite
often. I’ve also blogged about learning
from mistakes, mistakes are important. Be
humble in the way you interact and listen to people. People who believe they are never wrong, suffer
defeat far worse than those of us that recognize it as part of growing.
If I was wrong about the passwords from 2000 to 200x I
wanted to know. I wanted to understand
why, and what we could do to fix it. Had
I been wrong, I would have sung it from the mountain tops. I would have blogged on it and gladly advised
people on it. Instead I found the answer
I was looking for and the logic to back it up, I share that just as freely as I
would have my failure.
When everyone is fully committed to an idea that you do not
agree with instead of assuming they are wrong, consider that you are. It changed my point of view on the project. It allowed me to embrace a team that wanted someone
to take them on the path towards success and be part of that team. It also reinforced a very simple fact, I am
not always right and I need to listen to others.
In each case you have to want to achieve the end goal, and
that has to be more important than being right.
As always Dear Reader, thanks for stopping by!
Thanks,
Brad