"Who Gave YOU the Authority to Make Changes to MY Database?"
I was very new to a DBA position, and this was the question posed to me by a very, very, very angry user. “I’m duf SeQL DBA”, I not so eloquently replied swallowing was seemed like a very dry bite of a sandwich, I was enjoying before this exchange began.
Let me step back a couple weeks. I had just started a new position as the Sr. SQL DBA, the job was not shaping up like I thought. A disaster that occurred before I landed on the ground had wiped out all of the Documentation. I mean, there was zero documentation, there was no list of the servers to be managed, in many cases there were no environments other than production, minimal change management tracking, and the where to start list kept shifting. Also on my first day that my boss had taken me to lunch to warn me that the last 2 people to hold my position had been unceremoniously fired within 6 months? Initially I had been told I would only manage 5 Servers. That seemed like a cake walk, and to be honest I was a little disappointed. The number 5 was given because there was no list of servers, and these were the 5 most important servers that anyone could think of. Soon that list grew to over 100 servers all to myself.
I had been told of the utter importance of one particular system. So I figured that would be the best place to start. There were no regular backups of all databases, just some databases. Some that had an acceptable level of data loss of no more than 5 minutes, where in Simple recovery model and had not been backed up….ever.
So I went through the system following change management, documenting everything, setting up regular maintenance, getting everything in line, examining existing jobs, and gathering baselines. So when the user in question showed up, I had reviewed the jobs, looked at what they actually did, and changed the name of one job to better reflect what it did.
Unbeknownst to me, because it was not documented, the job I renamed was part of a business process another user would follow, set off manually, and because I changed the name (about 1 hour before this user would kick off this job) there was no job that matched the name exactly, as per what they had done before and in this particular Division that had started a storm that head straight to my desk.
There were several issues that were brought to light by this incident, 1st the lack of documentation would only lead to more issues, the process I was following for change management needed to be better defined, and I needed to know more about my databases.
The incident as it occurred involved the intervention of management, explanations on my part, and assurances that I had done everything right and thoroughly (which is probably why I continued to have a job), did I mention that on my first day that my boss had revealed to me that the last 2 people to hold my position had been unceremoniously fired, and this was before all of the documentation had been lost?
FIX IT! FIX IT! FIX IT!
FIX IT! FIX IT! FIX IT!
An Experience like this makes you check Monster will leave you with two alternatives, You can have a poor attitude to the customer in question, or you can acknowledge that things need to be fixed and do your best.
” Or as Oscar Rogers would say
Step 1: Fix
Step 2: IT
Step 3: Repeat Steps 1 & 2 TILL IT’S FIXED!
One of the things I did was re-define the Change Management process for Databases, meeting with the users and documenting every last detail became crucial, as well as developing baseline standards and making sure that they got implemented across the board, but more important engaging the users before and after the changes were made.
My main issue with interacting with the business was that the Database Administrator position had been such a source of instability that the business was not used to having a DBA, and in order for me to do my job I needed to re-introduce the profession and assure the business that I wasn’t “messing” with their data but it was my job to protect it, help them, and explain the how’s and why’s where possible. There was a lot of doubt and my issue was to build trust.
It wasn’t an easy task, but every time I was able to provide documentation, Server lists, or any Standard Procedures I was able to chip away at that doubt. Every time a disaster occurred, and we were able to recover it chipped away a little more.
Trust Comes Full Circle
A day eventually came when this user had an issue with their database. It was a case of logical corruption, the process they ran via human interaction requires some parameters. The Parameters that had been entered were incorrect. And it caused the data produced by the batch process to be incorrect and inconsistent. To compound when the data was mis-entered the user had waited to the point that rolling back the transactions, simply finding the most recent data entered, or recovery from backups was no longer possible.
A couple of things had changed since out last interaction, I had documentation on the process; I had created development and staging environments, and I also knew how guarded this user would be with their data so I wasn’t caught off guard.
The first thing I did was to re-ensure them that their data’s integrity was my top priority. I laid out the plan to correct the data (which was easy to find thanks to the documentation), I walked them through the change process, and how we would validate and test in each environment. Now these are not big revolutionary concepts and are pretty simple and standard, but when you go from not having structure to having a more standard process the difference is night and day.
By the time we had finished the user who had once surprised me while I was eating lunch surprised me by emailing my boss, his boss, and their boss to inform them what a great job I had done for them.
It was a very hard won victory, to get this user not only to trust me, but to trust that as a DBA I cared just as much about their data as they did. You can have issues with the business where they are slow to adapt technology, slow to move in a certain direction, or have misconceptions about technology as a whole. But if the business doesn't trust you to do what is best for them, then all the knowledge in the world won't help you sell your point of view.
A day eventually came when this user had an issue with their database. It was a case of logical corruption, the process they ran via human interaction requires some parameters. The Parameters that had been entered were incorrect. And it caused the data produced by the batch process to be incorrect and inconsistent. To compound when the data was mis-entered the user had waited to the point that rolling back the transactions, simply finding the most recent data entered, or recovery from backups was no longer possible.
A couple of things had changed since out last interaction, I had documentation on the process; I had created development and staging environments, and I also knew how guarded this user would be with their data so I wasn’t caught off guard.
The first thing I did was to re-ensure them that their data’s integrity was my top priority. I laid out the plan to correct the data (which was easy to find thanks to the documentation), I walked them through the change process, and how we would validate and test in each environment. Now these are not big revolutionary concepts and are pretty simple and standard, but when you go from not having structure to having a more standard process the difference is night and day.
By the time we had finished the user who had once surprised me while I was eating lunch surprised me by emailing my boss, his boss, and their boss to inform them what a great job I had done for them.
It was a very hard won victory, to get this user not only to trust me, but to trust that as a DBA I cared just as much about their data as they did. You can have issues with the business where they are slow to adapt technology, slow to move in a certain direction, or have misconceptions about technology as a whole. But if the business doesn't trust you to do what is best for them, then all the knowledge in the world won't help you sell your point of view.
Excellent post! And that last sentence is absolutely the key...trust! Great story, and congratulations on winning-over the business rather than just becoming another failure in the passage of DBAs through that company. You have not only changed their expectations and working relationship with you, but you have raised the bar everywhere every one of those people works in the future, because they have experienced how it SHOULD be done.
ReplyDelete