Dedicated hosting and SLA

When you are on a project with a stipulated SLA (service level agreements), like 1000 concurrent requests per minute, it is always good to be selfish.

That is, to hog all the system resources to yourself and avoid shared hosting.

What happened recently was that we had a client who asked us to develop an application and host it for them, with a certain SLA on response time. However, the client would also like the server to host another project.

We did not think much on it. However, at regular intervals, our application would slow down, with long response time. We wondered if we had to tune our application, web server, or databases.

Turned out however, that the other project hosted by the client would, at regular intervals, eat up almost 100% CPU usage for a short while!

That would surely hurt our stress test.

So we informed the client, shut down the other project, before doing our stress test.

Lessons learnt? Always be selfish and go for dedicated hosting. You might not meet a reasonable client every time.

Beware of firewalls

I'm always weary whenever I see firewalls in deployment architecture. It almost always spell problems. Problems that one will never realize during user acceptance tests. Simply because a user acceptance test environment will never mirror production environment. Hardware firewalls are not cheap.

Luckily, most firewall problems I had seen always boil down to the same thing.

Closing of idle connections.

Most applications would use connection pools, to reuse existing opened connections, for better performance. It can be for database connections, ldap connections, or any other kind.

However, a problem occurs if a connection is allowed to lie idle over a period of time.

Especially if the pool idle connection timeout is longer than the firewall idle connection timeout.

What usually happens is this:

1. Firewall idle connection timeout is set to 30 mins.
2. Database idle connection timeout is set to 60 mins.
3. Database connection timeout is set to 10 mins.
4. Connection is opened from client to server.
5. Connection is left idle for 40 mins, and the firewall closes the idle connection.
6. Client application retrieves the closed idle connection from pool, and sends a database query.
7. Client application blocks for a 10 mins before timing out, because the connection is already closed.

There are three ways to resolve this problem.

1. Set the connection to 'ping' the server while idle from time to time. This could be a select 1 query while idle.
2. Set the database pool idle timeout to be a value lower than the firewall idle connection timeout value (if you are allowed to)
3. This is the more common approach, which is to set the firewall idle connection timeout value to a value higher than the database pool idle timeout. This sounds exactly the same as the 2nd way, but people are usually more inclined to modify firewall settings than application settings.

Do not assume what a product officially support without first checking

Just a simple tip here. Never ever assume that just because a product has a new release, that it will have the same list of supported platforms and 3rd party products.

It is common for new releases to drop support for older platforms. Or less popular platforms.

Even if you are asked if a popular platform is supported, it is best to still look the list up. Check if there are any need to first patch the system or products. Find out all possible incompatibilities and known issues,

Always look up what is officially supported by the product company. After all, if a product is officially supported you know who to turn to if there is a problem. Otherwise, you are on your own resolving the issue. And you will really wish you had quoted a premium price for helping to support an unofficially supported product.

Meeting: pre and post activities

Ever went to a meeting with the clients/vendors with no idea what to ask, discuss, or report?

Ever left a meeting without being sure of what was achieved?

Or worse, not invited to the meeting itself, and not informed before and after the meeting at all on what was discussed?

These meeting are usually missing two important activities. The pre-meeting activity, and post-meeting activity.

Before meeting up with the clients/vendors, or any other external parties, it is always good and crucial to meet up internally first. Establish a set of understanding of where everyone's progress/standing is within the team, so that when meeting the external parties, a unified front can be shown.

In additional, the project manager (from this side of the team), could collect all the questions that needed to be asked to the external party before hand. List of issues to bring up, and any progress since the last meeting could be consolidated as well. Having this list prepared before hand is helpful. Popping the questions on 'anyone has anything to say' constantly in the meeting, or doing a round-the-table talking does not show that the project manager is in control. Probably would reduce the external parties' confidence in the team too.

Do some 'paper-play' as well before the meeting, especially on the issues to bring up. Think about how to response to the enquires, and offer solutions even before they ask for. This adds to the image of being professional. Definitely will justify the money spent on the team and project.

After the meeting, hold another internal meeting again. Go through once again what was discussed. Broadcast the decisions made and discussions out to the team if they were not in the meeting with the external parties. Their work require them to be aware of what is currently happening.

Also, always come away from the meeting with an action list. It is pretty impossible to not take any actions after any meeting. The team should own at least some actions, while the external parties would own others. Come back and start distributing the actions out too. Revisit the list from time to time, and report the status of this actions on the next meeting with the external parties.

Any meeting with the external parties without doing the pre and post activities would, in my opinion, not be effective, and one would end up with a confused, demoralized, and disgruntled project team.

Workflow solutions are never easy to do well

As I do more and more workflow related projects, a similar problem kept surfacing again and again.

Basically, the idea is this. A workflow, for example, approval for a new hire changes could go on for days. The request could route from departments to departments. It could sit in the manager's inbox for days before he acts on it. Now what if, while waiting to be processed, the hiring process changes?

In a paper world, that could be handle relatively easily. If there are more signatures needed, simply find the relevant parties and ask them to sign accordingly, on the same paper request.

However, most workflow systems cannot handle this well. In fact, most of them require a resubmission of the same copy of request! That is because these systems emphasis on the workflow, rather than the data flowing through the system. A rigid workflow system proves to be significantly harder to adapt to new changes 'in-place', applying the changes to the existing running requests.

Another problem I encountered was more interesting. Most approval workflows treat the request as pure data. There had been rare cases of items requested during submission to be no longer valid when the workflow actually has to act on the data (eg creating a login account on an in house system). This is especially possible for scheduled tasks, where you actually plan for usage in the future, either activation of new accounts or deactivation of existing ones.

Another interesting issue but non-technical has to do with the presentation of information during request submission. Let's say that you have a company going for a role based access policy. A manager role would have access to system A and system B. When a user request for a manager role, he is told that he would be given accesses to system A and system B. Now, while the approval workflow is going on, the policy changed. A manager role will instead only have access to system A. The right business implementation would do just that. Give access to system A. But the user might not be informed of the policy change. All he sees is that he is missing access to system B. Nothing wrong technically, but just some confusion. The workflow had never planned to inform the requester on the changes.

And another thing I keep seeing is the need to view the history of a request. What a user requested, when was it approved, when it was granted, etc. These are really request specific, and so each action must be audit manually. Luckily, the implementations I had been involved had mostly forsee these, and planned them in early. But there are cases where we had to modify the workflows to capture even more data beyond the requirements, as the client think up more and more of what they need.

Of course, most of these are beyond the responsiblities of the workflow engines we use. The solution provider should really plan to resolve these as part of the implementation. And they should be planned early. They are not easy to solve, but it will surely make for a much better user experience.

Application Internationalization and Localization

First off, a definition of what these are.

An application usually can and will be used by people from diverse cultural and language speakers. And so there will be demand for the application to speak to them in their own culture (date and money formats) and languages.

And so the act of creating culture and language specific versions is known as localization. But before that, you had to perform internationalization to extract all possible.

I have done two larger projects that involve internationalization of applications, with various smaller ones, but only one that actually involve localization.

And they sure are a pain to do. They use very simple concepts. Simply extract all string from the existing application and put them in a property file. And define a format string for each money and date format use, and place these in property file as well.

But if you treat each string as a unique entry, you tend to end up with alot of duplicated string. A message like "Approver name" could appear in both a form submission page, and a form status page.

It is extremely tempting at this point, to simply allow the two placeholders to use the same string entry. And in fact, that's a perfectly reasonable and valid approach.

Until the day come with you gotta change the text themselves. Imagine that, someone decided to use the message "Choose an approver" in the form submission page. Before the change to the message, he has to first evaluated the impact of change. Like, how many other parts of the UI is using the same string entry?

It is only when there are no other parts of the UI using the same string, could the developer make the change safely. But if it is actually used n many other areas, he gotta duplicate the existing entry, give it a new message id, and update that only.

So be careful of merging and reusing messages. Do them only if both are the same message, within the same context. A message string that is used during the selection of an approver, and one that is used to display the approver might have the same string, but they are in different context.

Another point in hand, is to really hire someone who is proficient in both the language translating from and to. The least worry you want to have is on the grammer and spelling related mistakes.

And lastly, avoid concatination of messages. You might have a message that goes like this: "Your account expired on XXX. Would you like to renew?"

It is tempting to have two string here, "Your account expired on ", and "Would you like to renew?". Do that, and you might wish you had never been involved in the project when there are message changes. One day the message might be changed to "XXX: Account expired. Renew now?"

It is better and cleaner, instead, to use a single string with placeholders like this: "Your account expired on {0}. Would you like to renew?". That way, you get the absolute flexibility of having the formatted value anywhere in the message, and you get to keep the message as a whole. During localization to other languages, the context of the message is clearer to the person working on it as well.

But granted, what I have described are simple principles and techniques, but doing the actual work itself is never easy. Especially when you are doing a 'extract message id' along the way you develop in a team environment. "I need a new message, does it exist? I have no idea. I'll just add it and hope someone will use it later". And then down the road the same message, of the same context, might have like five entries, all used in different part of the UI, just because they were all added by five different people.

So lastly, I proposed the addition of a internationalization owner, who owns, manages, and gives out message id. Tell the person the message you want, and the context, and the person decides if it should use an existing message id, or a new one.

I have not seen it in work, but it sounds feasible. Any comments?

Have an effective bug tracking system

Seriously. A bug tracking system is essential. I don't mean a software application. Just a process will do.

When a bug is reported, detailed steps of how the bug occurred should be provided. Screenshots would be helpful. The actors involved should be named as well. For example, for a request workflow, state the requestor, assignee, appover, and so on.

And of course, if this is a testing phase, try to have multiple test accounts. When a bug occurred, stop using the test account involved in the bug. This allows the developer to look into the data state when the bug occurred.

Though of course, in most cases that could not be done.

Back to the system. A bug will go through various phases. It will be open, fix in progress, testing, and closed, for the basic status list. It should be noted that the bug should always be closed by the group who reported the bug, not the developers themselves. This removes any future conflicts between the client and the vendor.

So, we have a 'new' bug. The user submits it with a description of what he was trying to do, what happened, and what SHOULD happen. I feel that it is essential that all three such things are submitted. Many times I encounter vague steps of what a user did, and what happened. And then they leave out the part on what should happen. So I have no idea which part of the happening was the bug.

A priority level is tagged with each bug as well, ranging from fatal (cannot proceed on and is a show stopper) to minor (work but is an annoyance). Most people use a number system, from 1 to 4, and depending on preference, 1 could be the fatal bugs.

It is likely that there will only be critical and fatal bugs. The user would usually want everything to be changed till it suit their taste, and if no one manages the defect list, be prepared to be sucked into a neverending whirlpool of changes. Negotiate and discuss the priority of issues with the user. Give and take for some of them. Most users are reasonable, but there are always those few unreasonable ones.. especially when politics come into play.

At times, rather than doing a time and effort consuming fix, a workaround can be suggested. Update the bug status with the work around suggestion, and throw it back to the user for review.

Now for the case where a fix is needed. The bug should belong to a single developer at any time. And this is actually where I wanted to complain at the start of this post.

The bug was originally assigned to someone else. But as that developer was assigned with too many fixes, the others started to help out. But without consulting and discussion with the original assignee, bug fixes get duplicated. Time and effort are wasted. Confusion arises.

Do a reassignment of defects with the original owner around. Please.

To carry on with the discussion of the system. After a bug is fixed, it should go through a round of internal test. The bug status should be updated accordingly. After it is verified to be fixed, it should be thrown back to the user for testing, and then readied or production.

This is another point of a bug fix cycle to be careful about. Sometimes, the fix expose another irrelevant bug. And the user would report that as part of the current bug, and keep the bug open.

Avoid that! As far as the facts are concerned, the reported bug was fixed! Any furthur bug or changes should be reported as a new bug or change request. This is especially important if the team is committed by contract to fix all bugs reported before a deadline, which is considered a milestone completion. Some clients might actually wish the milestone to not be reached, to avoid payment.

In conclusion, do take care into setting up a proper bug/defect tracking system. It could turn a profitable project into resource hogger and waster (if these are real words).

Avoid doing updates as delete/insert

Recently, for a project, we had interactions with a database schema in the following form.

User has multiple Positions, which has multiple Roles.

Now, here, we tried to do the quickest way out, thinking that, when a single entry in Role has changed, or been added, we simply did the following:

1. Delete all Positions of User
2. Delete all Roles of User (Role had a column entry of User ID as well)
3. Reinsert all Positions of User
4. Reinsert all Roles of User.

Why did we do that? Well, quite simply, because we cannot tell from the each object entry if it is an update or insert, and no way to tell deletion since the consumer of such an object actually remove the role from the position array to indicate a removal.

An alternative that sprang to mind was to do a select again before the updates, and do delta compare before updates. That seemed like an awful lot work to do.

So this delete/insert approach works.. but it proved to be a wrong approach.

It creates excessive and unnecessary strain on the database, in terms of redo logs. Depending on how you configure the database, the redo logs might be considerable, especially if you get hit with a 'power user' of say, 300 positions with 200 roles each. Sure, unreasonable example, but if some logic went wrong somewhere...

And also, such insert/remove bumps our primary key id generated value up significantly fast. I have no idea what happens when it reach the maximum limit, if there is one.

And finally, because the primary key id changes so very often, another way has to be introduced to uniquely identify each entry. It could be another running sequence number, or a composite key of a few values. You cannot rely on the primary key id, which becomes somewhat redundant.

A compromise could have been reached.

It might have been better to mark an entry with 'tags'. Eg, a role without an id indicate an insert statement. A role with an id and a 'isUpdate' to true is an update. While a role with an id and a 'isDelete' to true indicate a deletion. Rather than work with it transparently as removing objects from an array, use flags to indicate the operation to perform on each entry. This is more work on the consumer of the object, but better than the alternatives.

Allow for business policy changes

I was reading over Robert Martin's PDF on design problems and solutions to accounting software. I have yet to finish the PDF, but the problem they described was so familiar to me in some ways.

In fact, it is to do with policy/rule changes. Along with scheduled tasks.

Case in point. Two companies had a contract between them, the sale of a particular material at say $x per piece. Payment is usually after delivery, and there is a scheduled delivery two months later.

One month later, and the company signed a new contract, at $y per piece. So, when the materials are delivered, which rate should it be using?

It is entirely likely that the answer is, it depends on what the business wants for that particular contract. For some companies it might be $x, others at $y.

So when we develop such a system, we must first take care to not bind the delivery to a fix rate of $x.

On the other hand we should not bind it to the current value from the contract too. These are such major decisions that the system should avoid making.

In such a case, it would help to escalate the issue to a business user, informing them of the situation, and the choices.

Such a system is of course much more complicated, but it is likely what businesses want. They want automation, and the ability to intervene.

But remember though, for every intervention and decision, it should be recorded and audited, as part of the request history for future reports and audit purposes.

There are of course, many other similar cases where the variables of a system when a task is scheduled is very different from when it is to be executed. Email sending, account creation and deletion, role assignment, etc (mostly in the context of identity management since that is where I'm concentrating on).

But the solution are similar. Avoid binding values at schedule time. Verify that all is the same at execution time. If any differs, leave it to the business users to decide. They may decide to sell at a loss now to earn more later. The system can never advise or decide that.

Meeting: Focus on problems the right way

When working on a particular problem, there are many times when we will be slapped with a particular restriction that prevents us from progressing forward in an ideal way.

When we looked at this from the outside, we are able to rationally decide that the logical solution here, is to work around the restriction, such that we are still able to progress forward, even if it is slower. Or sometimes, suggest to get rid of the restriction.

Yet, when we are part of the working team, we stumble, running around in circles, not going anywhere.

Imagine this scenario. You are in a meeting. You encounter a restriction. You report this to the others in the team. All you get is constant grilling on why this happen. And then asking for a solution. Then various brainstorming of solutions. Which many would not make sense, or unachievable. And then the meeting end without even a resolution. And then come the next meeting, and the restriction is reported again. And then comes the same round, and some finger pointing and blaming game.

This is highly destructive to the team dynamics, as restrictions or problems might go unmentioned simply because.. mentioning it does not help at all. Stabbing begin as people try to shift the focus to other people's problem than one's own. The grilling lowers a person's confidence. And of course, most importantly, nothing gets done. Meeting is essentially useless.

One of the critical use of meeting is to bring everyone up to date with what's happening in the team, and harnass the collective intelligence and authority of the team to resolve stumbling blocks of the project. And the above scenario does not do any of those.

A strong project manager is essential, to groom the right culture for meetings.

When face with a restriction, all the team need to know is, how the restriction would affect solving the problem. A restriction is not the problem.

Remember, the member had never intended for the restriction to be there. It was not his fault. It is a problem, and not HIS problem.

And lastly, but most importantly, a mindset shift. Stop focusing on what cannot be done (the restriction). Focus on what could be done. Review your available options. And how those options could help to work around the restrictions or solve the problem.

Look into ways to remove the restriction, or if impossible, work around the restrictions. If need be, change or review the original solution to the problem. Change the original solution if all else fails.

This does not only apply to work, but in life as well. Sometimes we are so struck with what we cannot do, we do not progress, and forget about what we could do instead.

Be positive when faced with problems or restrictions. Focus on what we can do, not what we cannot.

Single point of configuration

In a software project, especially java, one would employ various third party libraries to ease and speed up development.

However, every library would likely have it's own configuration file or approach to configuration.

Let's look at a typical java project. A web based project would have a web configuration file (XML). It is highly likely to use logging, maybe configured with a property file. For sure there is a database connectivity, and since there is password involved, you might want to define it in the application server. And if you are using Spring, you might have a bunch more XML files to configure.

So that is like at least 3-4 files to configure.

Now let's say you are a very modular person. You like to have things configurable too. So you had a configuration file for the LDAP settings, one file for the security policies, one file for default email templates, one file for mail servers.

Now, imagine that a new developer comes along, and need to update the configuration of a bunch of things. The changes would be spread among various files.

Or imagine a fresh deployment, which require the configuration of a database connectivity, LDAP, email server. 3 files instead of one. Or maybe it is not a new server for production, but just a test server.

I was brought to the wordpress system. To configure a new copy of wordpress, all one need is to configure just one file. The file would contain all the settings. I did not look into it, but it woul also make sense for plugins to expose their settings in there too.

A conclusion I draw from this? Try to centralize an application configurations at a single location. Try to make it a facade for all the subsystems, even though they might be third party plugins, libraries, or systems. This helps in management of settings, and reduce learning curve for new developers or users.

Making audit data friendly

Auditing is a facinating topic to me. So much so that I'm writing another blog post on it.

So, let's assume that we have a UI screen that allows a user to update his particulars. And once the user submits, the data are pushed out to be updated.

The logical thing drawn here is that there should be an audit entry indicating the user initiated an update. Simple, easy. So far so good.

Now, what if the update looked simple, but was not? The user might have updated his name, password and telephone number. And assume that the password has to be pushed to an external legacy security system. The telephone has to be pushed to the company address book system. The name to the company address book system, as well as the HR system.

Each update could fail independently of each other. And since it actually push updates to external system, we really should audit them too.

So now we actually have four audit entries. One for the user clicking submit, one for the secuity system update, one for the telephone sytem, and one for the HR system.

But how do we know the other three updates belong to the same user initiated update action?

The problem gets worse when you had another background workflow that does similar update. So you had a system initiated update, a user initiated, and a bunch of interleaved updates. They could be on the same user, or different. The focus here is the audit log. Imagine how painful it would be to decipher which update belong to which action!

We need to group all related actions. In the example above, the user initiated action could generate an action transaction id. And all the sub-actions should be tagged with the same action transaction id. This allows us to view and perhaps even search the audit log by action transactions, giving us greater control.

And as a random thought, perhaps audit entries could be catagorized with detail level too.

Let's extend the update example furthur. Assume that an approval workflow is involved. There will be audit entries for request submission, approvals, escalations and resolution due to timeout, provisioning, and so on.

We could have a level of audit for request workflow. These are request submissions, approvals, escalations, etc.

Next, a level for provisioning. The target systems being updated, and the data changes.

Depending on what one want to view, they could view a 'business action' audit, or a 'technical' audit.

This is something I noticed missing in the few systems customization project I worked in. And we usually had to build this in. And I always wonder why they never think to build this in.

I'll probably be still pondering on auditing in business applications. I hope to be involved in creating an audit friendly application development someday.

Audit on XML data

I had to implement a feature recently for a particular requirement. Due to the nature of the product, a particular value had to be serialized into XML data before being persisted to pass through a particular adapter layer code. And just before persistence, the code would deserialize the XML data so that they can be written to the corresponding database tables.

The database is not storing XML data, to be clear. And the reason we had to do the serialization and deserialization is because the boundary of the adapter code accept only int, boolean, string, and list of strings. We needed list of objects. So our only escape route was list of strings.

It was definitely not pretty during auditing.

There is an in-built auditing feature. When data pass through the adapter, all old and new values are audited.

But now that the data are in XML, we get well.. a long string of XML. It was not too clear which value actually changed. One had to copy both old and    new XML string to some text file, do some formatting, and then do a diff on them.

The application was rather smart. Before passing the values to the adapter, they would first do thier own diff. If there were no changes, the adapter would not receive the values. The audit would not log these unchanged values too.

This, however, pose a problem with my xml data at times. Sometimes, the original value came out with a particular order of the attributes. And when saving, the order might change (perhaps I added/removed, or there was something different in the serialization). The attribute values remain the same, yet the order change, resulting in the application thinking that a change had occurred.

Imagine the frustration when you had done a diff on two values, and realized that the values were the same!

Auditing at the level of xml string is a bad idea. It always has been, and will be. If it was not for the in built functionality, and limited time, we probably would have reworked that part of the application.

Someone has to own the project

Working on the most recent project was a huge pain.

As mentioned, it was half on-site and half off-site.

The pain was beyond communication. It seems that at every moment, there is always something new to do.

And these are not new or changing requirements. They are hidden in the current code, unimplemented.

The lead on the project love to use a UI driven approach, or top-down as some would call it. Mock up the UI with dummy API calls. Nothing wrong with that.

Except when I was going to start changing some UI to use a new logic/design, I notice that some of the data the UI was displaying were hardcoded dummy data!

Now, you might think that time was saved, since I was going to update it to the new UI. But what was worrying was that when I was going to update this, I asked question on this part, and the lead was unsure if it was displaying data from the backend or otherwise!

It was never clear who owned a module. Sometimes a few of us would be working on the same library. It was never clear to me whose duty end and whose begins. And this might result in unimplemented logic.

And like many project, the requirement is constantly changing. And many of the business logic are still convinently locked in the mind of only a few. There is constant fear that not everything is implemented.

All I need is the confidence that someone has everything under control. He knows what had been done, and what not. And he can decide when something is done. That's what we lack.

Sounds like a good project manager is needed.

Working offsite

My earlier post mentioned that I had been on an overseas assignment.

There was this week, where I had the luxury of working offsite.

Except, it was no pleasure. It became more like a nightmare.

I had no ability to checkin my work into their subversion repository. I am constantly in the fear of being out-of-sync. On the second day we had a resolution. They would check out the entire repository. I would merge my code. And then send it back to them. I have no idea how the other co-worker working offsite would be doing.

And of course, for communication, we had google group talk. But it felt slow and unresponsive. I could understand. When I was on site, the google talk client could not work. I had to use the browser based gmail. And I would not be notified of new chats unless I switch over to check it.

And for me, communication was a huge problem. Because the requirements were not fix, and comprehensive enough. There were many doubts, and many things to clarify. I felt my productivity go down to a snail. So much so that while waiting for responses... I was really out of tasks to do. I went to read a book I had just borrowed from the library instead.

There was also the timezone problem. Thailand is an hour behind Singapore. And they usually start work at around 930. Which means 1030 for me in Singapore. And they work late. Well not too late nowadays. Around 1900~2000 Bangkok time. And that would be 2000~2100 for me in Singapore.

Which is why, after the second day, I started working from home. No point sticking to the Singapore timezone.

I do envy how 37Signals got their style to work. It did not work for me.

Software Developer Career Path in Singapore

If you are a software developer newly graduated in Singapore, what are your possible route of career advancement?

This is actually a topic that has been a constant frustration for me.

Outlined is a natural progression of a typical software developer in Singapore:

1. Developer -> Team Lead -> Project Manager.

Short and simple to understand. Most software developers would follow this natural progression, and is probably the only model they are familiar with. Project Manager is where the money is, which is of course essential for our day to day expense, especially in this country of high cost of living (I think we are ranked like 10? in the world).

Notice also that in this progression, a person develop less and less, but manage more and more.

Now what about those who do are not good in managing? Or uncomfortable in managing project and people?

In the current situation, they will just... try to manage. This path is a very natural model here, as I stressed. And in fact, for some, this is seen as progress, or promotion. It is just NATURAL. If you don't go up the chain, you are seen as well.. incompetent. Irregardless of how beautiful and perfect your code is. Or how knowledgable. Or how good at debugging you are. If you don't go up the chain, you are simply SECOND CLASS. Your pay and bonus will definitely be lower, because there is likely a cap at each stage you are in. Even worse, maybe the highest pay a developer can get will still be lower than the lowest pay a Team Lead get. Simply because a Team Lead manage, which means additional responsibility (this is true). But even if he suck at managing, he can probably still get more pay than the most component developer in the company.

But actually, there is an alternative route of progress, which is not that common in Singapore.

2. Developer -> Senior Developer (-> Architect) -> Consultant
*(-> Architect) represents an optional progression

The developer could be not good at managing, or simply uncomfortable with managing. This represents an attractive alternative route. All experience, built up since the time of development, are still utilized. In fact, the developer is free to, and strongly encouraged to, engage in active development at every stage of progress. Even when he is an Architect.

In this other route, the skills of a developer is capitalized and maximized.

But why is this model not popular in Singapore?

The answer is very simple. Singapore Companies do not recognize software development as a high quality skill. This arise when you have management who have no experience in the software development industry or domain, who had never done any software development before. They see software developers as expandable works, who can be easily replaced as and when. Every software developer, to them, has the same amount of output, and contribute evenly to the development of a project. Software developers can be added to a project anytime, which can result in a directly proportional speed increase.

Of course, we cannot deny the contributions by our Software Development Industry itself on this misconceptions. Recognizing Software Development as an engineering discipline gives people the perspective that the development lifecycle is predictable, and easily controlled. The original waterfall approach does not help, but it is widely adopted (agile and scrum is the right approach, but still need some time to properly displace waterfall in most companies). And then, the cheap outsourcing alternative is not helping as well. To the management, why pay 2x for a senior developer, when you can hire 2 junior developer? And of course, 4 foreign developer for that matter?

Which is why as software developer, we should recognize which route we are comfortable with. And identify companies that support the selected route.

IBM, Sun, and other MNC would be more keen in supporting both route of advancement, while most local companies like ST, NCS, would be more keen in supporting route 1. I am actually lucky to find a local company that support route 2. But still, the best approach is definitely to work overseas ;)

Stop anticipating requirements

As developers, we want an easier life. We worry and fear of unstable client requirements. We think of any possible ideas the client might come up with, and implement hooks for them in our current code (as long as they are not too tough, and even that is relative to a developer's skill).

Half the time, we are right, and we felt good about saving ourselves a lot of hell by putting in the hooks early.

The other half, the requirements never come. But we still felt good because we WILL be saving ourselves a lot of hell by putting in the hooks early.

But in general? The extensions and hooks we introduced made the system more complex. Code does not read that naturally, hidden within implementations of Template Pattern, nested one within another. The runtime structure of an application is so remotely different from the static structure of an application (class diagrams), that documentation no longer seem to be useful. In fact, they seem to be talking about totally different applications! The growing unnecessary hooks only add to more documentation, which seemed to scare every other developer away except yourself. And we wonder why.

But what could be worse? Well, that the requirements for a new feature came. And it is totally in conflict with the hook we planned. Well, we could argue that the hook was not well designed, and some change would be necessary, and then we could wait for the 'real' requirements to come. So for now, we add a hook to the hook, happy that we have added even more flexibility. And while we are at it, we thought of another possible client requirement. So we add another hook too.

And the story goes on. Hooks on hooks on hooks. on hooks. Half of them probably would never be used.

So what exactly went wrong? Obviously, the problem is us.

We tried to act like we are the client. Except we are not. We are probably the worse candidate to act like the client, since in most cases we are developing for a problem domain of which we are neither expert, nor even remotely familiar with.

Which is why as developers, we should always keep our solution simple. A simple solution is a flexible solution. Add hooks only if the actual client requirement comes. The value added service we provide here is faster delivery of solution, instead of a monolith and extensible solution but would require months to add new features.

Remember, in most cases the clients are the domain experts, not us. They may not always know what they want, but they do know what they do day-to-day. Spend time with them, understand their need. Do not give in to their desires. And do not give them something that they do not need.

The next developer is probably not as good

Of course, this is an unfair judgement call, since you probably had not even met the next developer.

But this is something my colleague kept trying to hammer into me, which has its valid point.

Why does he say so?

Well, simply because my code is really, rather hard to dive into.

We are actually working on an in-house project, where we want to set up a simple framework, and have the future development 'outsourced' if possible. We had a quick stint with an intern, but it took really long for him to get into.

There are various technology used.

Spring DI. Spring JDBC. Spring LDAP. Spring JavaMail.

Apache Quartz. Apache Struts 2. Apache Tiles. Apache Velocity. Apache Maven.

Logback. JUnit. Mockito. Selenium.

All rather simple to learn and use... to me that is.

He made an interesting point. That most developers do not learn new stuff on their own. They are component with Java, but perhaps as a language. They do not go experiment with the bleeding edge. They are comfortable with the basics. Some of them are probably still in the plain old Servlet and JSP era! Annotations? What are those? How do we use them?

I came initially from the 'productive rock star' perspective. Anything that keeps the stack lean and mean, and allow me to quickly deliver business values, the better it is.

But given his perspective, I honestly doubt a newcomer can easily pick it up. I will have to explain Maven, set it up for him. Explain unit testing and Selenium. Mocks. What Spring DI is. Where are the Struts localization files. How Tiles work. Logging with logback. Struts interceptors. Struts validation... blah blah blah.

Which brings me back to reiterate the same point in a different word from him again. Assuming that there are component developers in the company. But being in a company, there are definitely programmers of different level and standards. Being on the bleeding edge might also mean that the knowledge of using it being constraint to the 'elite'. The rest 'inferior' developers might find it harder to pick up. Or maybe even impossible and never.

And as we saw in the phrase, "It is all very much good to place all decision making in the hand of the perfect man, but what if the perfect man has a bellyache?"

We are using Java. Why Java? Sure, it takes more line to do the same work in Java than probably in Ruby or Scala. But it is 'clear and simple', and 'mass-market'. It is very much easier to get java developers here than the others.

So right now I'm taking a step back. Annotations might have to go. Various Struts specific concepts might have to go too. An abstraction layer of the above mentioned stack might have to be developed, such that we can have a consolidated framework and usage pattern, along with documentations on how to use that.

That was his idea, but it definitely has merit. Given a one to one mapping (even though it is a thin redirection layer), we can simply tell the incomer to use 'our libraries', complete with tutorials and example. It promotes a standardization of company code, and probably is much more helpful in the long run.