Programming: Feeling burdened

Recently I had been giving much thoughts into the programming languages like C++ and Java. I mentioned only these two because those are the ones I had been working in most of the time.

I felt somewhat burdened.

When the project grows beyond a certain number of classes and methods, it felt like a pain trying to even look at the source code. Even though the files were structured to be pleasant to read, there is this weird growing sense that something's not right.

I remembered Uncle Bob mentioned in his book, 'Clean Code', that the sequence of methods placed in the source file matters. It had to flow right. Perhaps that is my main problem.

I tried to keep my public methods as a block, and my private methods as another block. It did not work, since you had to jump from the public down to private when understanding code. Then I had all related private methods right below each public method. However, now, at a glance I cannot know what all the public methods of a class are (except through perhaps a smart IDE filter).

This is just wrong! Why are we still concern with ordering of methods in a source file! We should be working on a higher level of abstractions on them! We should be looking at only method blocks where they matter! The IDE should be smart enough to filter them out!

(I still miss VisualAge for Java from IBM).

But that's not all that made me feel burdened. I feel slow when developing on these. Perhaps its not the languages themselves. It could be my approach. I am very much a TDD guy. But recently, much affected by all the readings from 37signals on Getting Ready, etc. I wonder. Perhaps I am over TDD? Or are they even necessary?

Even Kent Beck, in his recent post on his JUnit Max, confessed that he did not take a TDD approach. There are plenty of code that were not test-covered.

It seemed that, in a rush to deliver, we had to sacrifice test coverage.

But of course, they do not apply to all scenarios. The reason I am feeling these are because, well, when I am working on my own hobby projects. I felt that I am slow in crunching out and moving towards the goal.

Lastly, a part of the burden is due to the language (definitely). The syntax are really to verbose. Lack of closures, excessive need to declare private variables with accessors. Constructors which simply set values. Lack of named parameters, which result in the need of parameter objects. There are just no lack of complains.

Perhaps I should move to a lighter, more agile language. Maybe even dynamic language.

Working offsite

My earlier post mentioned that I had been on an overseas assignment.

There was this week, where I had the luxury of working offsite.

Except, it was no pleasure. It became more like a nightmare.

I had no ability to checkin my work into their subversion repository. I am constantly in the fear of being out-of-sync. On the second day we had a resolution. They would check out the entire repository. I would merge my code. And then send it back to them. I have no idea how the other co-worker working offsite would be doing.

And of course, for communication, we had google group talk. But it felt slow and unresponsive. I could understand. When I was on site, the google talk client could not work. I had to use the browser based gmail. And I would not be notified of new chats unless I switch over to check it.

And for me, communication was a huge problem. Because the requirements were not fix, and comprehensive enough. There were many doubts, and many things to clarify. I felt my productivity go down to a snail. So much so that while waiting for responses... I was really out of tasks to do. I went to read a book I had just borrowed from the library instead.

There was also the timezone problem. Thailand is an hour behind Singapore. And they usually start work at around 930. Which means 1030 for me in Singapore. And they work late. Well not too late nowadays. Around 1900~2000 Bangkok time. And that would be 2000~2100 for me in Singapore.

Which is why, after the second day, I started working from home. No point sticking to the Singapore timezone.

I do envy how 37Signals got their style to work. It did not work for me.

Software Developer Career Path in Singapore

If you are a software developer newly graduated in Singapore, what are your possible route of career advancement?

This is actually a topic that has been a constant frustration for me.

Outlined is a natural progression of a typical software developer in Singapore:

1. Developer -> Team Lead -> Project Manager.

Short and simple to understand. Most software developers would follow this natural progression, and is probably the only model they are familiar with. Project Manager is where the money is, which is of course essential for our day to day expense, especially in this country of high cost of living (I think we are ranked like 10? in the world).

Notice also that in this progression, a person develop less and less, but manage more and more.

Now what about those who do are not good in managing? Or uncomfortable in managing project and people?

In the current situation, they will just... try to manage. This path is a very natural model here, as I stressed. And in fact, for some, this is seen as progress, or promotion. It is just NATURAL. If you don't go up the chain, you are seen as well.. incompetent. Irregardless of how beautiful and perfect your code is. Or how knowledgable. Or how good at debugging you are. If you don't go up the chain, you are simply SECOND CLASS. Your pay and bonus will definitely be lower, because there is likely a cap at each stage you are in. Even worse, maybe the highest pay a developer can get will still be lower than the lowest pay a Team Lead get. Simply because a Team Lead manage, which means additional responsibility (this is true). But even if he suck at managing, he can probably still get more pay than the most component developer in the company.

But actually, there is an alternative route of progress, which is not that common in Singapore.

2. Developer -> Senior Developer (-> Architect) -> Consultant
*(-> Architect) represents an optional progression

The developer could be not good at managing, or simply uncomfortable with managing. This represents an attractive alternative route. All experience, built up since the time of development, are still utilized. In fact, the developer is free to, and strongly encouraged to, engage in active development at every stage of progress. Even when he is an Architect.

In this other route, the skills of a developer is capitalized and maximized.

But why is this model not popular in Singapore?

The answer is very simple. Singapore Companies do not recognize software development as a high quality skill. This arise when you have management who have no experience in the software development industry or domain, who had never done any software development before. They see software developers as expandable works, who can be easily replaced as and when. Every software developer, to them, has the same amount of output, and contribute evenly to the development of a project. Software developers can be added to a project anytime, which can result in a directly proportional speed increase.

Of course, we cannot deny the contributions by our Software Development Industry itself on this misconceptions. Recognizing Software Development as an engineering discipline gives people the perspective that the development lifecycle is predictable, and easily controlled. The original waterfall approach does not help, but it is widely adopted (agile and scrum is the right approach, but still need some time to properly displace waterfall in most companies). And then, the cheap outsourcing alternative is not helping as well. To the management, why pay 2x for a senior developer, when you can hire 2 junior developer? And of course, 4 foreign developer for that matter?

Which is why as software developer, we should recognize which route we are comfortable with. And identify companies that support the selected route.

IBM, Sun, and other MNC would be more keen in supporting both route of advancement, while most local companies like ST, NCS, would be more keen in supporting route 1. I am actually lucky to find a local company that support route 2. But still, the best approach is definitely to work overseas ;)

Audit and Security are not meant to be interceptors

I was working on an administrative web interface for a user management product, and was trying to find out how best to do audit and security.

I should throw up the conclusion first: Audit and Security are not meant to be interceptors.

And now for the journey to that conclusion.

We are in the year 2009. Java annotations are on the rampant. Every problem looks like a nail to that hammer. And it is no surprise that there are many projects which implement audit and security annotations, and run them through interceptors like AOP.

Which I tried as well. It worked great initially. Marking methods to be auditable and securing methods with roles were simple, and does not obstruct code logic.

Until the security got more 'realistic' and requring the audit to be 'useful'.

In the original concept, a user could be an administrator, and able to modify any users. In the updated requirement, a user could manage multiple group of users. No longer that simple to just check by a single role a user has. The role would be dynamic too. To support multiple dynamic groups after deployment, the role name would be built up from the name of the group of users. And annotations for role name support only strings that can be determined from compile time!

Granted, a more complicated security annotation might work. Something along the line of annotating a method with a security policy instead of a role name...

But let's move on to audit first. The most common audit approach is probably marking methods as auditable, and then record all input arguments and output result of that method.

That seemed to me to be a rather... technical audit rather than a business audit trail. Translation of the audit information to business information is not so easy. And there are definitely times when the required audit information might not be part of the input arguments nor output result. And this might violate the SRP to some people, but it's entirely possible that a single method is in fact, multiple events and require two audit entry instead of one.

My point here is probably this, that for a given business scenario, there are definitely clear requirements on what audit information to capture. And these information usually go beyond the simple 'method invocation with arguments, results, or exceptions'.

I'm definitely not against annotations and interceptors. They are excellent idea, and work great for many cases. My grip is that audit and security functional requirements are never that trival, in the sense that many out of the box interceptors will fulfill the requirements. And there is just too much hype in audit and security interceptors as the holy grail.

Danger of being overly focused (to a skillset)

I started playing with XCode, Objective-C and iPhone Application Development recently (which was about 3 days ago).

Some background on my skillset first.

I started out learning Pascal and Foxpro, and then proceed to C++ with Visual C++. Along the way I picked up VB, and then Java. Then I worked with Java professional for about 3 years, and then out came C# (within that 3 years), which I then decided to make a jump for it, but ended up with VB.Net and Visual Studio for about a year. And then I decided that Java had a better career opportunity back then. So it was back to Java and Eclipse for another 2+ years.

And this is where I am now. Which is to say that I am strongly comfortable with Java and Eclipse (but seriously, Eclipse is an excellent IDE).

During the time I read up on Ruby and Python, but resisted developing with them, blaming the lack of an IDE with the quality of Eclipse.

And now that I want to try out iPhone Application Development, I had to pick up Objective-C and XCode.

I started reading up on both topics. And with every read, Objective-C felt like an excellent language. Named parameter was a big plus for me. Categories were a good way to slice up code into separate files (much like .Net's extension methods). Other than that, much was the same as old time (to me). XCode itself was a good tool, with code completion support (though I have yet to read anything on refactoring, and not much books talk about unit testing though they are available online).

And then came the time of actual usage. I was confident that with my experience in C++ I could tame that Objective-C beast. And the tool did seem good.

I was wrong. Not on the account of Objective-C. That was pretty easy. I am a firm believer that language syntax are simply decorations. And so I had concentrated on honing the more important skills of writing clean code, design patterns, building stronger foundation. But development still take a while, because I was so used to Java. Coding in Objective-C requires some mindset changes. Recognizing new symbols.

XCode was a much bigger killer. Code completion works, but code 'suggestion' was a different key. I had to retrain my instincts. I have yet to figure out unit testing, but probably will complain when it get working, citing it as different and weird compared to Eclipse (but that is definitely due to my overly used to Eclipse, not because of XCode). Checking for compilation errors was different and had to get used to. And looking at the logs while debugging was... well just wonder why I had to switch my view in XCode. Perhaps there is a different project layout view (I was sure I read something on it), and I should try that.

So why so much pain? I became overly reliant on a single tool and language. And that is definitely a very dangerous thing. Regardless if you read up on different tools or languages, if one does not actually try using them, they will not be able to operate under different environment. And as developers, we MUST be as flexible as possible. One can never know what tools one might use in the future of their career. (I am currently cursing on using Visual Studio in Windows XP VM on my Mac... GRRRR).

And after years of using Java, a garbage collecting language, I had to revert back to a non-garbage collected language. More care is put into each line, and there is always a wondering thought on if I should release the memory of something returned from a system call, or not if I am returning them as return arguments. It takes some getting used to.

I probably should have tried out Ruby and Python more often, and C++ too, with a variety of tools. Life would be better.

Stop anticipating requirements

As developers, we want an easier life. We worry and fear of unstable client requirements. We think of any possible ideas the client might come up with, and implement hooks for them in our current code (as long as they are not too tough, and even that is relative to a developer's skill).

Half the time, we are right, and we felt good about saving ourselves a lot of hell by putting in the hooks early.

The other half, the requirements never come. But we still felt good because we WILL be saving ourselves a lot of hell by putting in the hooks early.

But in general? The extensions and hooks we introduced made the system more complex. Code does not read that naturally, hidden within implementations of Template Pattern, nested one within another. The runtime structure of an application is so remotely different from the static structure of an application (class diagrams), that documentation no longer seem to be useful. In fact, they seem to be talking about totally different applications! The growing unnecessary hooks only add to more documentation, which seemed to scare every other developer away except yourself. And we wonder why.

But what could be worse? Well, that the requirements for a new feature came. And it is totally in conflict with the hook we planned. Well, we could argue that the hook was not well designed, and some change would be necessary, and then we could wait for the 'real' requirements to come. So for now, we add a hook to the hook, happy that we have added even more flexibility. And while we are at it, we thought of another possible client requirement. So we add another hook too.

And the story goes on. Hooks on hooks on hooks. on hooks. Half of them probably would never be used.

So what exactly went wrong? Obviously, the problem is us.

We tried to act like we are the client. Except we are not. We are probably the worse candidate to act like the client, since in most cases we are developing for a problem domain of which we are neither expert, nor even remotely familiar with.

Which is why as developers, we should always keep our solution simple. A simple solution is a flexible solution. Add hooks only if the actual client requirement comes. The value added service we provide here is faster delivery of solution, instead of a monolith and extensible solution but would require months to add new features.

Remember, in most cases the clients are the domain experts, not us. They may not always know what they want, but they do know what they do day-to-day. Spend time with them, understand their need. Do not give in to their desires. And do not give them something that they do not need.

Towards Better Code: No side effects, shorter methods, exit early, and log every invocations

Recall earlier that I had mentioned that I learnt through logs of existing program/framework.

It's weird, but I cannot emphasize enough about the importance and need to log.

I have been using a rather new approach to logging. Well, not exactly to logging, but a new approach combining logging and coding style.

First off, I keep my methods short. For example, given the following algorithm:

public void login(final String username, final String password) {
  // authenticate
  // check if user container is active
  // check if user is active
}

It will most likely evolve into the following code:

public void login(final String username, final String password) {
  authenticate(username, password);
  verifyUserContainerIsActive(username);
  verifyUserIsActive(username);
}
void authenticate(final String username, final String password) {
  final boolean valid = checkUserCredentials(username, password);
  if (!valid) throw new AuthenticationException();
}
void verifyUserContainerIsActive(final String username) {
  final Container container = getUserContainer(username);
  if (container.isInactive()) throw new UserContainerInactiveException();
}
void verifyUserIsActive(final String username) {
  final User user = getUser(username);
  if (user.isInactive()) throw new UserInactiveException();
}

That's just a rough idea. Basically, each method is composed of many more methods, which serve to describe what each step do. Benefits? Eliminate the need to comment complicated algorithms. Comments run the highest risk of being out of sync with the actual code intent.

Also, notice the use of exceptions in the inner methods. I had started out with an alternative idea, that the inner methods would return true/false values, which would. It basically create the following nasty code block:

public void login(final String username, final String password) {
  if (isValidCredentials(username, password)) {
    if (isUserContainerIsActive(username)) {
      if (isUserIsActive(username)) return;
      throw new UserInactiveException();
    }
    throw new UserContainerInactiveException();
  }
  throw new AuthenticationException();
}

Of course, the better way is as follows:

public void login(final String username, final String password) {
  if (isNotValidCredentials(username, password)) throw new AuthenticationException();
  if (isUserContainerNotIsActive(username)) throw new UserContainerInactiveException();
  if (isUserIsNotActive(username)) throw new UserInactiveException();
}

Personally, I still felt that my first example up there produce a much more readable version of the algorithm, which communicates the intent much better.

But the basic idea/rule here is, exit early. What is most important is to communicate the basic intent of the code. The other possible cases are, well, exceptions (pun intended), and should not clutter the original intent of the code.

And finally, on to logging.

Now, I log all entry and exit of a method, as well as method arguments and method return values. Ideally this should be done with AOP, but I am not doing that yet.

What benefit does this logging bring me?

I can actually, from tracing the method entry logs, form a very clear view of how a request pass through the entire system. An example of the above log might be as follows:

enter: login {username:kentlai, password:xxx}
enter: authenticate {username:kentlai, password:xxx}
exit: authenticate
enter: verifyUserContainerIsActive {username: kentlai}
exit: verifyUserContainerIsActive
enter: verifyUserIsActive
exception caught: UserInactiveException

With shorter methods, logging every method invocations, I can trace the flow very accurately. Sure, there is an increase in the size of logs, but the return in such investment is easier debugging. Without even hooking up to the debugger.

With the logged method arguments, and knowing which method is causing a problem (from the logs), I can actually write a quick unit test to fix any potential problem found.

But there is one last magic elixer in this combination. No side effects.

Basically, each class should not have mutable state. The only mutable state should be from external storage (eg. database). And if a function need to use any values from such storage, they should, themselves be encapsulated into methods of their own, with the return values logged.

Why? This allow you to put together a complete picture, the context and environment of which a method was executed under. When every variable becomes known and can be supplied once again to a method during unit testing, a method can be guaranteed to work consistently and constantly as expected, which allows for more robust code.

Funny it took me so long to realize such principals.