Single point of configuration

In a software project, especially java, one would employ various third party libraries to ease and speed up development.

However, every library would likely have it's own configuration file or approach to configuration.

Let's look at a typical java project. A web based project would have a web configuration file (XML). It is highly likely to use logging, maybe configured with a property file. For sure there is a database connectivity, and since there is password involved, you might want to define it in the application server. And if you are using Spring, you might have a bunch more XML files to configure.

So that is like at least 3-4 files to configure.

Now let's say you are a very modular person. You like to have things configurable too. So you had a configuration file for the LDAP settings, one file for the security policies, one file for default email templates, one file for mail servers.

Now, imagine that a new developer comes along, and need to update the configuration of a bunch of things. The changes would be spread among various files.

Or imagine a fresh deployment, which require the configuration of a database connectivity, LDAP, email server. 3 files instead of one. Or maybe it is not a new server for production, but just a test server.

I was brought to the wordpress system. To configure a new copy of wordpress, all one need is to configure just one file. The file would contain all the settings. I did not look into it, but it woul also make sense for plugins to expose their settings in there too.

A conclusion I draw from this? Try to centralize an application configurations at a single location. Try to make it a facade for all the subsystems, even though they might be third party plugins, libraries, or systems. This helps in management of settings, and reduce learning curve for new developers or users.

Making audit data friendly

Auditing is a facinating topic to me. So much so that I'm writing another blog post on it.

So, let's assume that we have a UI screen that allows a user to update his particulars. And once the user submits, the data are pushed out to be updated.

The logical thing drawn here is that there should be an audit entry indicating the user initiated an update. Simple, easy. So far so good.

Now, what if the update looked simple, but was not? The user might have updated his name, password and telephone number. And assume that the password has to be pushed to an external legacy security system. The telephone has to be pushed to the company address book system. The name to the company address book system, as well as the HR system.

Each update could fail independently of each other. And since it actually push updates to external system, we really should audit them too.

So now we actually have four audit entries. One for the user clicking submit, one for the secuity system update, one for the telephone sytem, and one for the HR system.

But how do we know the other three updates belong to the same user initiated update action?

The problem gets worse when you had another background workflow that does similar update. So you had a system initiated update, a user initiated, and a bunch of interleaved updates. They could be on the same user, or different. The focus here is the audit log. Imagine how painful it would be to decipher which update belong to which action!

We need to group all related actions. In the example above, the user initiated action could generate an action transaction id. And all the sub-actions should be tagged with the same action transaction id. This allows us to view and perhaps even search the audit log by action transactions, giving us greater control.

And as a random thought, perhaps audit entries could be catagorized with detail level too.

Let's extend the update example furthur. Assume that an approval workflow is involved. There will be audit entries for request submission, approvals, escalations and resolution due to timeout, provisioning, and so on.

We could have a level of audit for request workflow. These are request submissions, approvals, escalations, etc.

Next, a level for provisioning. The target systems being updated, and the data changes.

Depending on what one want to view, they could view a 'business action' audit, or a 'technical' audit.

This is something I noticed missing in the few systems customization project I worked in. And we usually had to build this in. And I always wonder why they never think to build this in.

I'll probably be still pondering on auditing in business applications. I hope to be involved in creating an audit friendly application development someday.

Audit on XML data

I had to implement a feature recently for a particular requirement. Due to the nature of the product, a particular value had to be serialized into XML data before being persisted to pass through a particular adapter layer code. And just before persistence, the code would deserialize the XML data so that they can be written to the corresponding database tables.

The database is not storing XML data, to be clear. And the reason we had to do the serialization and deserialization is because the boundary of the adapter code accept only int, boolean, string, and list of strings. We needed list of objects. So our only escape route was list of strings.

It was definitely not pretty during auditing.

There is an in-built auditing feature. When data pass through the adapter, all old and new values are audited.

But now that the data are in XML, we get well.. a long string of XML. It was not too clear which value actually changed. One had to copy both old and    new XML string to some text file, do some formatting, and then do a diff on them.

The application was rather smart. Before passing the values to the adapter, they would first do thier own diff. If there were no changes, the adapter would not receive the values. The audit would not log these unchanged values too.

This, however, pose a problem with my xml data at times. Sometimes, the original value came out with a particular order of the attributes. And when saving, the order might change (perhaps I added/removed, or there was something different in the serialization). The attribute values remain the same, yet the order change, resulting in the application thinking that a change had occurred.

Imagine the frustration when you had done a diff on two values, and realized that the values were the same!

Auditing at the level of xml string is a bad idea. It always has been, and will be. If it was not for the in built functionality, and limited time, we probably would have reworked that part of the application.

Someone has to own the project

Working on the most recent project was a huge pain.

As mentioned, it was half on-site and half off-site.

The pain was beyond communication. It seems that at every moment, there is always something new to do.

And these are not new or changing requirements. They are hidden in the current code, unimplemented.

The lead on the project love to use a UI driven approach, or top-down as some would call it. Mock up the UI with dummy API calls. Nothing wrong with that.

Except when I was going to start changing some UI to use a new logic/design, I notice that some of the data the UI was displaying were hardcoded dummy data!

Now, you might think that time was saved, since I was going to update it to the new UI. But what was worrying was that when I was going to update this, I asked question on this part, and the lead was unsure if it was displaying data from the backend or otherwise!

It was never clear who owned a module. Sometimes a few of us would be working on the same library. It was never clear to me whose duty end and whose begins. And this might result in unimplemented logic.

And like many project, the requirement is constantly changing. And many of the business logic are still convinently locked in the mind of only a few. There is constant fear that not everything is implemented.

All I need is the confidence that someone has everything under control. He knows what had been done, and what not. And he can decide when something is done. That's what we lack.

Sounds like a good project manager is needed.

Programming: Feeling burdened

Recently I had been giving much thoughts into the programming languages like C++ and Java. I mentioned only these two because those are the ones I had been working in most of the time.

I felt somewhat burdened.

When the project grows beyond a certain number of classes and methods, it felt like a pain trying to even look at the source code. Even though the files were structured to be pleasant to read, there is this weird growing sense that something's not right.

I remembered Uncle Bob mentioned in his book, 'Clean Code', that the sequence of methods placed in the source file matters. It had to flow right. Perhaps that is my main problem.

I tried to keep my public methods as a block, and my private methods as another block. It did not work, since you had to jump from the public down to private when understanding code. Then I had all related private methods right below each public method. However, now, at a glance I cannot know what all the public methods of a class are (except through perhaps a smart IDE filter).

This is just wrong! Why are we still concern with ordering of methods in a source file! We should be working on a higher level of abstractions on them! We should be looking at only method blocks where they matter! The IDE should be smart enough to filter them out!

(I still miss VisualAge for Java from IBM).

But that's not all that made me feel burdened. I feel slow when developing on these. Perhaps its not the languages themselves. It could be my approach. I am very much a TDD guy. But recently, much affected by all the readings from 37signals on Getting Ready, etc. I wonder. Perhaps I am over TDD? Or are they even necessary?

Even Kent Beck, in his recent post on his JUnit Max, confessed that he did not take a TDD approach. There are plenty of code that were not test-covered.

It seemed that, in a rush to deliver, we had to sacrifice test coverage.

But of course, they do not apply to all scenarios. The reason I am feeling these are because, well, when I am working on my own hobby projects. I felt that I am slow in crunching out and moving towards the goal.

Lastly, a part of the burden is due to the language (definitely). The syntax are really to verbose. Lack of closures, excessive need to declare private variables with accessors. Constructors which simply set values. Lack of named parameters, which result in the need of parameter objects. There are just no lack of complains.

Perhaps I should move to a lighter, more agile language. Maybe even dynamic language.

Working offsite

My earlier post mentioned that I had been on an overseas assignment.

There was this week, where I had the luxury of working offsite.

Except, it was no pleasure. It became more like a nightmare.

I had no ability to checkin my work into their subversion repository. I am constantly in the fear of being out-of-sync. On the second day we had a resolution. They would check out the entire repository. I would merge my code. And then send it back to them. I have no idea how the other co-worker working offsite would be doing.

And of course, for communication, we had google group talk. But it felt slow and unresponsive. I could understand. When I was on site, the google talk client could not work. I had to use the browser based gmail. And I would not be notified of new chats unless I switch over to check it.

And for me, communication was a huge problem. Because the requirements were not fix, and comprehensive enough. There were many doubts, and many things to clarify. I felt my productivity go down to a snail. So much so that while waiting for responses... I was really out of tasks to do. I went to read a book I had just borrowed from the library instead.

There was also the timezone problem. Thailand is an hour behind Singapore. And they usually start work at around 930. Which means 1030 for me in Singapore. And they work late. Well not too late nowadays. Around 1900~2000 Bangkok time. And that would be 2000~2100 for me in Singapore.

Which is why, after the second day, I started working from home. No point sticking to the Singapore timezone.

I do envy how 37Signals got their style to work. It did not work for me.

Software Developer Career Path in Singapore

If you are a software developer newly graduated in Singapore, what are your possible route of career advancement?

This is actually a topic that has been a constant frustration for me.

Outlined is a natural progression of a typical software developer in Singapore:

1. Developer -> Team Lead -> Project Manager.

Short and simple to understand. Most software developers would follow this natural progression, and is probably the only model they are familiar with. Project Manager is where the money is, which is of course essential for our day to day expense, especially in this country of high cost of living (I think we are ranked like 10? in the world).

Notice also that in this progression, a person develop less and less, but manage more and more.

Now what about those who do are not good in managing? Or uncomfortable in managing project and people?

In the current situation, they will just... try to manage. This path is a very natural model here, as I stressed. And in fact, for some, this is seen as progress, or promotion. It is just NATURAL. If you don't go up the chain, you are seen as well.. incompetent. Irregardless of how beautiful and perfect your code is. Or how knowledgable. Or how good at debugging you are. If you don't go up the chain, you are simply SECOND CLASS. Your pay and bonus will definitely be lower, because there is likely a cap at each stage you are in. Even worse, maybe the highest pay a developer can get will still be lower than the lowest pay a Team Lead get. Simply because a Team Lead manage, which means additional responsibility (this is true). But even if he suck at managing, he can probably still get more pay than the most component developer in the company.

But actually, there is an alternative route of progress, which is not that common in Singapore.

2. Developer -> Senior Developer (-> Architect) -> Consultant
*(-> Architect) represents an optional progression

The developer could be not good at managing, or simply uncomfortable with managing. This represents an attractive alternative route. All experience, built up since the time of development, are still utilized. In fact, the developer is free to, and strongly encouraged to, engage in active development at every stage of progress. Even when he is an Architect.

In this other route, the skills of a developer is capitalized and maximized.

But why is this model not popular in Singapore?

The answer is very simple. Singapore Companies do not recognize software development as a high quality skill. This arise when you have management who have no experience in the software development industry or domain, who had never done any software development before. They see software developers as expandable works, who can be easily replaced as and when. Every software developer, to them, has the same amount of output, and contribute evenly to the development of a project. Software developers can be added to a project anytime, which can result in a directly proportional speed increase.

Of course, we cannot deny the contributions by our Software Development Industry itself on this misconceptions. Recognizing Software Development as an engineering discipline gives people the perspective that the development lifecycle is predictable, and easily controlled. The original waterfall approach does not help, but it is widely adopted (agile and scrum is the right approach, but still need some time to properly displace waterfall in most companies). And then, the cheap outsourcing alternative is not helping as well. To the management, why pay 2x for a senior developer, when you can hire 2 junior developer? And of course, 4 foreign developer for that matter?

Which is why as software developer, we should recognize which route we are comfortable with. And identify companies that support the selected route.

IBM, Sun, and other MNC would be more keen in supporting both route of advancement, while most local companies like ST, NCS, would be more keen in supporting route 1. I am actually lucky to find a local company that support route 2. But still, the best approach is definitely to work overseas ;)