How to Manage Data Within an Agile Organization

Good data governance makes quality information timely available throughout the lifecycle of the organization. This should include aspects of availability, usability, consistency, data integrity, and data security. Governance should, however, adapt data processes to the needs of the parent agile organization.

That said, effective data management does not exist outside the rest of the IT process, or indeed the entire organization. It must obey the overall needs of the greater whole, for data cannot exist for data’s sake.

Data Permeates Every Aspect of the Organisation

We can compare a data management system to the cardiovascular process in our bodies, transporting oxygen and nutrients to organs and limbs. However, it is of diminished value if the rest of the body is unwell.

Hence data belongs to the organization and not for its own sake. We must do away with legacy strategies that are too slow and cumbersome to meet the needs of the agile organization. We must replace these with systems that are quick in responding to changes in the internal environment.

Moreover, our people have a right to access the information they need quickly, to perform their jobs to their full potential. Agility must permeate every aspect of a lively organization. Only then can a living organism be both stable and dynamic at the same time.

Litmus Proofs for Agile Data Systems

• Data must, therefore, evolve to suit the needs of the organization, so it can react quickly to fresh opportunities. New agile databases and spreadsheets must simplify migration to new data stores. This is essential in order to flourish in a competitive environment.

• Data systems, as we have said have no right to be agile for agility’s sake. They must optimize to meet the needs of an organization becoming agile. Of course, these refinements occur in parallel in practice. We just wanted to emphasize the fact the needs of the parent structure are the specification.

• The result must be ‘good enough not perfect’ because quality meets the stated needs of the customer, beyond which it surely becomes a self-seeking effort. We run the risk of delaying development teams when we forget Voltaire’s dictum “the best is the enemy of the good”.

• Agile is getting quickly to where we need to be. We can go back later and tidy production databases, if there are meaningful advantages for the bottom line. Data exists only to serve. It has no right to exist on its own.

• The customer owns their database and must be active in developing it. Data in IT is a service organization. It must actively involve the user in data system development. For in that way they will understand the system – having already tested it – when it goes live on the day.

• There are no silos in agile organizations although there are specialists. People involved in data systems must have done more than a cursory tour through the customer environment. They must understand more than just how it works.

Shared knowledge empowers effective cross-functional relationships before the need for change. Multi-function teams drawn from different departments understand requirements faster and develop new data systems quicker because they have the best of both worlds. Constant collaboration is the right vehicle for managing data.

• These multi-faceted teams are then well-positioned to develop data systems using test-driven development. This method holds it’s best to provide developers working specifications, as opposed to comprehensive documentation they will scarcely read. This approach also supports continuous improvements as bureaucracy morphs into support and enablement.

Process Goals of Agile Data Management

The litmus tests for agile data systems enable IT to migrate from self-limiting bureaucracy to a culture of enablement and support. The objectives of data projects become their benefits, as opposed to allowing the best to become the enemy of the good.

Goal # 1: Constantly Improving Data Quality

The agile community makes concrete quality techniques available off the shelf. Database refactoring, for example, involves small changes to achieve incremental improvements. Doing database testing and continuous database integration moves us closer to where we want to be without significant user disruption.

Goal # 2: Developing Data Assets

Data assets are the record of what we have achieved. There will be some legacy ones we can discard, while we must improve others to curtail spreadsheet risk. And protect our test data, application output files, databases, documents, and other critical systems from corruption. Data security is fundamental because we must restrict access to spreadsheets to those with authority. We also need a fast, effective way to extend these permissions when something changes in the parent organization that requires our immediate action.

Goal # 3: Redefining Data Structures

Our data structures must be fit for purpose, including arrays, files, records, tables, and trees. This allows us to find, and quickly manipulate information using the least possible processing time, memory space and bandwidth. That’s what agile is in a nutshell: quick and effective every time.

Streamlining Workflow within the IT Department

The IT department as a whole must contribute to achieving these agile data goals too.   Data management must seamlessly connect across to delivery teams, operations, and release management too. There is no space in agile organizations for silos and turf wars.

This may be one of the more painful transitions you have to make. Delivery teams must learn to acceptance guidance from release management, and turn to operations for help when they hit snags. They will find this very different from the traditional serial approach but they will welcome the change as they learn.

Collaboration across IT teams places greater value on individual interactions than on processes and tools. It encourages people to leave silos to exchange information, and discover the power of working together.

Bolstering Resources for Delivering Key Artefacts

Task # 1: Adapt / Develop Data Artifacts in Line with Company Strategy

Artifacts are things we create that are of value. In the IT sense, we think of metadata of common objects, entity types, and data elements. We also create master test data for critical entity types. It’s essential we don’t keep them for artifact’s sake.

Therefore, these database objects must serve, and contribute to overall company strategy. An astute data manager does not do this alone. They consult and collaborate to ensure all critical data artifacts are in place and aligned with the overall goals of the company.

Task # 2: Get to grips with the key business models and applications in user departments

These spreadsheets and other processes input into your data artifacts and are hence a key to your ultimate success.

Identify critical files across the organization. Once you understand the nature of these user systems, you can find collaborative ways to increase their business value, while ensuring access is reviewed regularly to minimize spreadsheet risk of corruption.

Task # 3: Build Muscle and Support into Delivery Teams

The best data management strategy is for nought without a delivery team to implement it. A thoughtful data manager invests time and effort in ongoing training. A more knowledgeable environment frees them up for more forward-looking, strategic tasks. However, they still need to keep a handle on progress in their delivery teams.

That’s because deliverers look to their manager for support when encountering design problems, or difficulties accessing legacy systems. The goal is always cooperation through collaboration. Managers do not belong in silos in agile spaces.

Task # 4: Keep a Steady Hand on the Tiller of Delivery and Ops

Implementing agile in IT does not mean abandoning proven conventions, for example, user experience feedback and data security. A watchful manager needs to make sure these conventions remain in place.

They should likewise work hand-in-hand with operations to ensure their production data sources are intact at all times. It should be possible to do most of this work on a dashboard from the office.

However, a data manager’s prime responsibility is ensuring the quality of data constantly improves. For if they do not manage this, it will surely slip-slide in the opposite direction. Management’s task is never done, no matter the quality of resources under its control.

How to Implement Agile in Your IT Department

• Expose the Truth: The old, non-agile ways of doing things may embed deeply in your IT department. You may have adopted this culture too. Challenge the effectiveness of your data management practices. Do this as a team.

• Obtain Buy-In: You can’t afford passengers on your agile journey. Refuse to accept the linear way of doing things. Support innovations that achieve your operational goals faster. Recognize the value of others’ opinions.

• Reward Achievements: Invest in training but insist on seeing the benefits flow. Encourage social interaction within the team and with users. Celebrate their successes, not your own. You are only a custodian.

Therefore, don’t hesitate to call for assistance from specialists if you lack the necessary skills within your own department, but insist that knowledge-transfer is part of the package. You can and will succeed when you have the resources you need and the wisdom to empower them.

However, only know this before you begin. Agile is a never-ending journey with a leader striding before, but never too far. Beware the day you think you have finally achieved agile. For this will be the day the old, bureaucratic way of doing things comes knocking at the door.

---

This post is a contribution from our friends from Clusterseven.com