Playing with elephants – Hadoop?

People who know me know that I’m interested in databases and SQL. I continue to describe myself as an Analyst rather than a Developer or DBA, but I also think it is useful to have a basic understanding of the characteristics of the tools one might be using. I’m busy at the moment but I thought that it was high time I “nailed my colours to the mast”.
I intend to start looking at Hadoop before the end of the year.
I know that may seem like a long way off, but it is getting closer all the time.

Here is one of the articles which grabbed my attention:

There is nothing like committing that you are going to do something as a bit of encouragement.

Why do I want to look at Hadoop?

  • First of all, I want to do it as an excuse to “brush up my Unix” (sorry Linux)
  • That makes it an excuse to buy and use some new (to me) hardware.
  • It will also be a reason to collaborate with an old acquaintance of mine.
  • And I find the idea of “map reduce” intriguing and trying it out is the best way to learn.

As I said, I’m busy, so that’s it for this week!

Day to day plans – Having a dynamic To-Do list

I’ve come to regard myself as being like Winnie-the-Pooh “a bear of little brain”. I like to give myself a task, settle down to it and get on with it without distractions. Sometimes the task itself can be quite complicated though.

In recent posts I’ve shared a view of “Strategic” and “Tactical” plans with you. I know you haven’t seen what is inside them but I find having them enormously useful. That brings me to the next level down: the day to day, dynamic To-Do list.

Having a dynamic list of the tasks I have to do is enormously useful. If I’m managing other people doing things, then I need to know what they are doing too. This is what the classic project plan is all about. Even a small team needs a shared plan. I used to create them using spreadsheets but now I use a special purpose tool.

The picture above shows screenshots from today’s dashboard. The tool I use is PBWorks Project Hub ( I use the “Freemium” version, and I find it adequate for my needs. It does most of the things I would have been doing with a spreadsheet and it does some other things too.
It is very good at doing the boring Project Office things like bugging people (including me) when they are due to be doing something today. It is also good for collecting reports of progress made or problems encountered. All that has to happen is that people have to get into the habit of making one-line notes on the task they are working on each day.
I have a geographically dispersed team. The two of us exchange notes and have the occasional phone call, but the project dashboard gives us something that we can go back to for status information without a lot of cumbersome bureaucracy (which would usually devolve to me!). The only downside of this approach is that we are using a cloud service which requires internet connectivity and could be vulnerable. That means that I take what I think are appropriate steps to have the critical information stored in some other form elsewhere.


What do you do when things go well? Make a new plan!

A little while ago I shared my “Tactical Plan” with you. Things have gone well! So well that some of the projects have been completed much sooner than I expected. Some of this is down to good luck and having estimates which were not much better than guesses. The question is: what to do now?
I expect we are all (painfully) familiar with the situation where a project is slipping behind the intended schedule. Sometimes the reasons are the same as the reasons for my success: luck, inaccurate estimates and sometimes “force majeure”. After the inevitable struggle to get back on track by cracking the whip, the response is usually to re-plan. That is exactly what I have done in response to my recent success.
Here is the redacted version of my latest tactical plan. In structure it is identical to its predecessor. The changes are in the bits you cannot see.
Let me tell you about what is in the updated plan (without giving any secrets away):
  1. The objective was not achieved last time (but that would have been a miracle). So that has remained unchanged.
  2. The “New Product” development is still (more-or-less) on track, so that remains unchanged from last time. Basically it is – “Keep working away on the new product!”
  3. The “Marketing” project from last time has completed. It’s objective was to create something new. That has been done. Now I have a new project to use what was created by that project as part of a regular activity. I still call it a “project” because it certainly hasn’t become “business-as-usual” yet. I hope it will do eventually.
  4.  The “Administration” project from last time has been completed. It was necessary, and it is making things work more smoothly, but it is complete and there is no need for follow-up.
  5.  The completion of the “Administration” project has created an opportunity to start something new in the “Marketing”. That has already thrown up some interesting ideas, but the Tactical Plan is reminding me to focus on my current objectives. The new stuff can be considered for the next plan.   
The tactical plan is part of the project wiki (I may show you a little more of that in the future). It is visible to all the contributors and I have a printed copy above my desk, a little to my left, as I type. It is great as a means of reminding us all (especially me) what we have agreed we are concentrating on.

Do we learn from our mistakes?

This is the time of the year when a lot of us spend a little time reviewing how well we did last year and what we plan to do this year. As you may have noticed, I’m making my plans a little more obvious this year. Before we get too committed to the planning process we should ask ourselves:
Do we learn from our mistakes?
Because if we don’t, or at least if we don’t try to learn from our mistakes, then the activity is essentially futile and we would be better doing something else instead.
I think that the IT industry is particularly bad at this. We have very little sense of history. We claim to be making progress – but are we?
Here is one person’s attempt to write down an outline of that history. I can’t say I agree with it entirely. In fact I haven’t really tested it critically. But do think that it is a worthwhile effort to present one view of that history.

Why do I think the IT industry is bad at learning from it’s mistakes?

Why do I think the IT industry is bad at learning from its mistakes? Basically, I think this because I frequently get a feeling that “I’ve seen this before”. Now there is no doubt that a lot of progress has been made. A lot of that progress is down to “Moore’s Law” which has enabled whole industries to sprout, bloom and flourish.
The reducing cost of hardware in general and processing power, storage and communications has enabled things to happen which while they were conceivable, were hardly practical just a few years ago. This blog and the thousands like it are an example of that.
But, if you look at various internet forums as I do, you will find recurring themes which I am going to share with you here and I may pick up as topics at some time in the future. The thing is that many of these meta-topics have been running for donkey’s years!
  • How do we write “requirements” so they are understood by the people who want the system and also by the people who are going to build the system?
  • How do we create things so that people get what they are expecting?
  • How do we estimate how long it is going to take us to do (practically anything)?
  • How do we manage a project so that it comes in “to specification, on time and on budget”?
  • How do we prevent “silly little bugs” creeping into the system?
Maybe you have some ideas for other topics in the same area, or maybe you have some solutions to some of these conundrums.

Wheels within wheels, plans within plans, but we do need action!

A week ago I shared the front page of the strategic plan for my business with you. Plans are all very well but what we really need is action, but of course we need controlled action which is moving us towards the objective.

With this in mind, I created what I’ve described as a “Tactical Plan” which I’m going to share with you. The tactical plan sets out an objective and some projects I am going to be working on for the first 3 months of 2015. I’ve included a redacted plan below. I first heard the word “redacted” in association with US Government documents. I think I would probably use the word “censored” instead. By-the way the Russian word for “editor” is “redactor” (редактор). I’m not sure if that is one of the little tricks laguages play on us, or whether it tells me anything about the Russian attitude to editing.

That really is all there is to the Tactical Plan. It is pinned to the wall in my office and it reminds me what I am achieving at the moment. It helps me to focus and keeps me from getting distracted. Of course, there are more detailed plans as well.

As you can see from the redacted plan, I have an objective, an end date and three projects. The three projects are addressing areas which I know need work: Product and Marketing are hardy perennials and in this case I felt it was time to do a bit of spring-cleaning in the thing I use for the detailed plans and tracking. I may show you that some time in the future.