Why KABI was created?

Context

Our primary goal was to build a BI solution for our division from scratch. A totally greenfield project. We also had some additional responsibilities such as;

• Support unavoidable high priority ad hoc report requests from various teams
• Support existing OLTP production reports including handling incidents 
• Support non-report incidents from time to time
• Production data analysis 
• Responding to RFIs/RFPs
• Third party BI supplier coordination

All that we had in place was a set of requirements that I had created before the team was formed based on analysis of existing reports, analysis of previous and on-going report requests, understanding source systems and data and envisaging what could be provided to management and customers based on previous experience and feedback collected from stakeholders.

Almost all of the development teams in the organization were working using agile process and scrum was the chosen methodology. Naturally, when we started forming a new team to develop the BI solution for my division we also started as a scrum team.  

Challenges

The team was new, 3 of the source applications production support team members joined our team, but they, just like me, were new to the tools (Pentaho Data Integration-PDI and MicroStrategy) that we had to use for development. We were asked to use same tools that another larger team (X) was using or will use in the future and also follow the same code framework for PDI developed by X.  So we had major dependencies on team X but unfortunately we were at the bottom or not at all in the list of priorities for X.  

As we had moved production support team members to our team, they still had to support ex-team on an unplanned basis as and when issues popped up there. Hardware, software and network requirements had to be ordered based on a high level idea and these would be provisioned in unpredictable timelines. Also, neither I nor my team members had experience in implementing a greenfield data warehouse / Business Intelligence solution using agile methodology.

Problems with Scrum in our Project

With all these unplanned activities and dependencies we were to work in scrum mode and provide sprint commitments every two weeks. We would spend hours discussing on the user stories, assigning story points, task breakdowns, estimations and capacity planning but at the end of the sprint we were making some progress but not up to my or other team members' expectations. And we were not able to achieve sprint commitment by large extent because of  lack of knowledge on the tools or due to lack of support from X or non-availability of required infrastructure (like severs) or because of unplanned production issues support or ad hoc report requests or people just going on unplanned leaves. There was also unequal work load distribution between team members.

The daily stand-ups was more of a formality. Retrospectives were partly good however wouldn’t result in any noticeable change as no two sprints were the same.  Last day of the sprint would be the sprint demo and some implementations would be totally different from what was expected. We would have planned user stories already for the next sprint two days before last day of the current sprint, now that would have to change because of tasks overflowing from current sprint to the next sprint or because of features not meeting requirements. In short, too much time spent in planning something which eventually would change during sprint and be reworked and hence less real work done. 

As a team we started committing lesser user stories/tasks for future sprints than we could actually do. My feeling about scrum was that it was too restrictive and not as flexible as our team should and could be and scrum involved too much of administrative tasks as it required lot of changes on the JIRA agile scrum board which I as a product owner had to keep up to date. We had to re-estimate some of the partially completed user stories and find how much capacity we had in the next sprint. I felt an immediate need was there to change the methodology so that we could spend more of our time on working and less on planning to work and thereby make better progress.

Introduction of KABI to the Team

I started reading about different agile methodologies and found that Kanban more or less fits what we required. A methodology that would still be visual, keep the user stories flowing through various statuses but wouldn't require too much of planning and administrative work and user stories doesn't have to fit into a sprint cycle. I intentionally wanted to customize it so that we could do what we needed to do and let no one tell us this is how Kanban should be done. I documented the customized Kanban and named it KABI and introduced this to the team. As if the team was waiting for something like this they were very open to give it a try. We started making good progress and all the while the team remained highly motivated. This was really a welcome change and we continued using KABI. 

For more on KABI refer to previous post - KABI - New Agile Methodology for BI