Junk Change and Pastiche Product Management

There is a malaise affecting Change Management and Product Management 

Junk - Photo by Jessica Palomo on Unsplash

Junk [creativity] : Worthless writing, talk, or ideas

Pastiche : An artistic work in a style that imitates that of another work, artist, or period

I have a problem with the way that modern software development methodologies are being exercised by a significant proportion of the practitioners in the industry.

So much value has been placed on performing the mechanics of Agile or DevOps processes that the value of those processes is being lost under a sea of buzzwords and procedural nonsense.

This is not about "Is Agile Dead?" or "DevOps doesn't work!", I don't want to get involved with that sophistry. What I want to do is point out an increasing trend towards the belief that just doing Agile or DevOps will bring the benefits. The benefits stem from the good engineering practices that these methodologies promote, through a degree of formality, language and organisation. Without recognising and cherishing the engineering behind the methodology, too many teams are burning the trust of their investors and business managers.

So what is Junk Change [Management]?

• being unable to say why a change is required;
• making changes that have no associated metric;
• making changes that are mainly aesthetic;
• making changes that have no relation to end-user/Persona activity;
• refactoring without a reward;
• making changes that test/affirm no hypothesis about your users;
• using non-experts to approve changes;
• making non-testable changes.

And what is Pastiche Product Management?

Pastiche Development - Photo by Jesse Roberts on Unsplash

• having a product vision that never gets evolved or referenced;
• having Personas divorced from the product plan;
• Planning things that don't need planning;
• Estimating things that can't be estimated;
• Grooming a backlog with opinions rather than teeth;
• having "statement only" Stand-up meetings;
• having sprint tasks for BAU activities;
• doing retrospectives that change nothing;
• allowing unchallenged slow cadence/velocity;
• condoning rubber stamp PR code reviews and release approvals;
• writing unremarkable release notes;
• having no method for user feedback;
• ignoring "difficult" bugs.

Knowing "Why"

Knowing why you are making a change and why that change is important is the single most influential factor in promoting productivity and healthy product development. Ask yourself "why am I changing the colour of this button?" and then continue to ask why all the way back to the point where you reach the raison d'être for the business. If there is no realistic chain of connections then you don't have a valid reason for making that change.

The same goes for the mechanics of a methodology, knowing why you have stand-ups and why you do code reviews is very important for the success of those steps in the process. And everyone needs to know the reasons so they can hold the management team to account when a retrospective changes nothing or there are too many cards on the Kanban board.
Collectively knowing "why" is the basis of a shared purpose.

Understanding Users and Business Models

Large consumer goods companies such as P&G and Unilever spend a lot of money researching their consumers in many ways including H&P (habits and practices) studies, trying to understand them. Senior executives sometimes even visit the homes of tame consumers to try and gain a degree of empathy for their lives and see first hand how they use the product. The more successful technology companies also have a very strong customer focus, for example Jeff Bezos said:

"Focusing on what customers want or need has driven many of Amazon's most profitable business moves."

Linking all changes to a vision that chases down a series of consumers' needs and wants is key to good product management. They may not know they need it or want it yet, but being able to strategically address those needs at some point in the product roadmap is something that needs to be clear. A good example comes from looking at Google's dominance of the mapping space, analysis shows that the product decisions were made many years ago with some big investments as Justin O’Beirne's recent blog post unravelled; the whole roadmap is aimed at providing a superior user experience eventually (7 years later). Perhaps Google Maps is just a stepping stone in a greater product vision.
The team need to understand the user and their needs, there are a number of viable mechanisms for doing this but the one that I find works best for software solution is the Persona and macro level Use-case documentation. The team also need to be aware of the corporate mission and where that intersects the users lives, as well as a high level view of how the company plans to turn that point of intersection into a formal business model. A lot of developers and developer focused product managers fail to see the connection between good user modelling (Personas and Use-cases) and feature development. Without an accurate internal representation of how the users and customers the majority of product decisions are simply guesswork.
The user model defines the way in which you can ascribe value to features and hence prioritise, groom and make key decisions about the product and its roadmap.

Feedback, Metrics, Releases & Product Testing

"It's About Feedback and Change" is the title of an Agile Software Development paper by Williams and Cockburn (from 2003); it was written at a time when Agile was not yet the dominant methodology for software development, and while the acceptance of Agile has matured the key pillars of the agile manifesto seem to still be missing from the daily lives of too many of its practitioners. Feedback is a valuable element of the information required to make meaningful changes to both the software and the process of creating that software.
Direct user feedback is great, but it comes with a cost - either the cost of soliciting it or the cost of processing the un-moderated stream. Metrics provide a way of interpreting user behaviour and ascribing the value associated with those behaviours, either from changes in user retention, user engagement or more direct positive outcomes (up-sell or social vitality). Retrospectives and team meetings also have a cost and so productivity metrics are also a part of the development information required for a healthy agile team (when used positively).
Feature flags and A/B testing are common ways teams release change; and this works well from a process decoupling point of view, where the act of completing the development and the exposure to the end users are not tied to the same release step. But these tools are designed to allow the developers to finish and move on to the next task, and allow the product owners to work out the impact of the changes.

Hard Problems, Big Bad Bugs & Feature Leader Marketing

Commercial advantage comes from delivering products and services that either give your users a feature advantage, facilitate a cost reduction or provide any other significant differentiating factor (e.g. GDPR compliance). The source of these advantages for a tech company usually comes from solving a hard problem, for example the audio fingerprinting algorithm that Avery Li-Chun Wang created to launch Shazam. If you are not solving hard problems and constantly moving the dial in any significant way, and your competitors are, then you will rapidly lose your marketability and commercial leverage. Writing the paper is only step 1 of the product development, you need to get the functionality developed and put it in the hands of users, learn from how they use it and iterate to create the commercial gain.
Sometimes the hard problem is not visible to the user, perhaps it is security or issues of scale, perhaps it is a nasty bug that creates the operational problem to solve; I call these "nice problems to have" because you only get them if you are doing most things right, and so shying away from these problems is commercial suicide - you need to tackle them head on!
It doesn't matter if you are operating in a new or mature market (e.g. mobile phones) the only difference is that the more mature the market the more well known the critical features are, but you still need to be addressing both the known and the unknown/unexplored features; the product only needs to be one step ahead on each one to be market leading from a feature point of view. In other words, the product doesn't need to be years ahead in one area, it needs to be slightly ahead in all areas. Incremental improvement of features will only take you to the point where it starts to get hard, by focusing on the hard problems then you give your product that extra feature step at each iteration.

Branding and Aesthetics

There is a valid place for the importance of branding and aesthetic changes in product management. But the investment in those areas needs to be proportional to the value it brings. Branding can be very subjective and often a personal statement, but its value is found in the generation of familiarity, trust and making your brand synonymous with the qualities of the product you want to promote.

Summary

Think:

• Is the methodology actually working?
• Do the changes matter?
• Does everyone know why?
• What features make the difference?

Spike Notebooks - Better Agile Discovery for Developers

Spike Notebooks - Better Agile Discovery for Developers

tldr; apply data science tooling to software development research spikes.

Problems & Rabbit Holes

I don't like it when my team come to me with a problem and then don't offer up a solution. Sometimes they can see what is going wrong, occasionally they think they have a idea of how to solve it. Probably worst of all they have no idea how long solving the issue is going to take. Another common pain point is the phrase "yes I have looked at it and ..." which normally means they have just read a blog post like this one ;) and are walking blindly into someone else's solution.
In an agile process (during backlog refinement) this blind uncertainty is problematic and your typical agile team will attempt to resolve this through a spike, which is "a small task done to reduce uncertainty about a larger task". It is usually aimed at finding out why, what or how the next step should be completed.
The issue that consistently crops up is that the work done on the spike is valuable to the organisation as a whole, as it is a form of research, and what often happens is that some of the results of the spike will filter into a README.md, code comments or a gist or pastebin (if you are lucky) - but the majority of the technical effort is lost in a developer workspace or a defunct code branch. There is no real way to merge the spike work into the whole project reliably.

I have seen developers that will happily march into the thick of the problem and just start "fixing" things in the hope that it all comes our right and proper in the end. But I also see those same developers getting stuck in the rabbit hole: and by this I mean that for example, in order to solve problem (a) they introduced library (b) which caused a version incompatibility with (c) whose update requires a monkey patch (d) which meant we had to introduce CI/CD builds for component .... and so on down the rabbit hole. In other words, directly solving the problem means solving lots of problems as a chain.
At the other end of the scale a developer might read a blog or the documentation and establish that the solution is theoretically possible. This might be enough if the docs and the software are reliable, but this plan will still carry risk because it has not been proven at the critical points - and it is this risk that will blow any estimate of the resulting implementation work.
Solutions that require architectural changes are particularly prone to the kind of problems described above, either spending too long spiking the architecture or not de-risking the architectural change enough.

What is needed is a way of demonstrating that a solution exists, proving the risky parts of the solution and showing how they fit together. This might involve some coding, perhaps stepping into a rabbit hole, but then being able to backtrack on that and take a different path without having to revert a branch or comment out great swathes of code - all the time preserving the discoveries made and the thinking behind it. The goal needs to be to gather enough information about the solution to make a firm estimate of the work to implement with minimal uncertainly.

Welcome Data Scientists

Over the last 2 years I have been running a variety of transitional development teams, teams of data scientists and mixed data+engineer teams. The data scientists have brought in new skills, new approaches and previously unseen tools to the world of software development. Jupyter Notebooks are one such tool. A developer might think of a a notebook as a collection of markdown documentation, executable code snippets and recorded view-able results (including graphs and images); here is an example.

Photo by FloorTwelve on Unsplash

So can you use notebooks for spikes? They need to be:

• Easy for the team to use: file CRUD, correct language
• Replicate enough production code to work for a spike
• Review-able by other team members
• Kept with the code, but not overlapping it
• Link-able, documented and demonstrable
• Isolated enough to run alternate versions
• Lead into clear decision making (e.g. LADR)
• Seed TDD with fixtures and case outcomes
• Supporting teamwork

Are notebooks easy enough to use?
Notebooks used to be hard to run and quite a learning curve, but that is changing. The Jupyter Docker Stacks project is making ease of use a reality. And efforts like Jupyter Lab are all heading towards better drop-in work environments. The following is an example of how to create the basic python3 jupyter lab on a developer laptop using docker (because you are all using containers now right?):

docker run -d --name jupyter -v /home/${USER}/workspace :/home/jovyan/work -e GEN_CERT=yes -e JUPYTER_ENABLE_LAB=yes -p 8888:8888 jupyter/minimal-notebook

Then you can add (or compose) your own language kernel. There are many to choose from covering JavaScript, Java, Kotlin, TypeScript, Ruby, Go, C# and obviously Python. If you are using R or Python ML/data science libraries (e.g. tensorflow) then you can get them by using a more specialised stack image without having to go through an extra install step. Scripting languages are more aligned to this form of development, so there are more hoops to jump through to get a compiled language working.

Can you work on the spike?
This really depends on how well your software is written and structured. The principle is that you pull in the code you don't want to change, write the code you do want to change in the notebook and then execute the code and/or tests to see the differences. The notebook magic %load can be used to pull in your existing code or snippets of it. The %%script -bg magic allows you to start supporting systems (e.g. databases), and the %run magic will start your program from its entry point. If you need to temporarily overwrite code you can use %save to replace or patch a file.

How does the team review the spike?
I expected this to be a straightforward step, but it is more complicated than you think. Notebooks are stored in JSON format with a lot of timestamps and binary blocks and so a typical code pull review system will treat them as diff-able, which means you have to always click through to view the original. You can export the notebook to PDF or markdown and review that, but that takes you outside the main code process. Github renders the notebook in a readable form, but it is a bit flaky on larger notebooks. Bitbucket requires a plugin to work. The online renderers like https://nbviewer.jupyter.org/ will only show public notebooks. This leaves you with 2 choices: 1) run a private nbviewer, or 2) use a browser plugin. We opted for the plugin because of an IT policy, and I am planning on releasing the plugin to the chrome web store soon. By sticking to a process that closely follows code review the existing agile steps still apply. Incidentally there is an option to export the notebook as reveal.js slides.

Where does the notebook live?
We have ended up putting one notebook file (.ipynb) next to the code for each question we are asking, and then have a notebook at a higher level to link them and represent the spike as a whole. This takes advantage of the markdown documentation feature, which can create hyperlinks between relatively addressed files. So, for example, a high level notebook might be .../Spike - How do we make database X globally available.ipynb, which might refer to .../terraform/myproject/Can we tunnel Redis across regions.ipynb and .../src/mycomponent/dao/Can we encode X data in Redis.ipynb; then you might also have .../test/What is the failure rate of packets over tunnels.ipynb; and finally you might add later in the process .../test/What is the cost of using GCP spanner.ipynb. This accumulates knowledge and experience alongside the code it is affecting. Notice that spikes are not about answering all the questions, just the critical ones and the most unknown, engineers should be able to accurately estimate the work to complete at the end of the spike sprint. When we re-structure we move notebooks to an archive directory so that they don't get buried in the git history. We aim to create new notebooks rather than change notebooks because of the problems merging large blocks of high-change JSON.

How does this help with the development process?
In the example above we have an overall unknown (summary), an infrastructure unknown (temporary infrastructure), a code unknown (temporary code) and 2 operational unknowns (graphs and charts). At any point someone can jump back in and re-run the notebook and see if the result changes - for example if GCP Spanner pricing changes. In theory you can use notebooks for tracking A/B testing, but that is a whole different conversation.
Notebooks not only provide a neat way of hanging on to the knowledge created during the spike but also provide a means to quantify the benefits and risks of a change proposal (e.g. it will cost $900/m to expand to Australia maintaining 3-9s error rates). These kinds of information would normally get estimated and end up in someones drive as a spreadsheet instead of being openly referable in the code base. It is this transparency that provides the key benefit of this approach.
It is straightforward to reference a spike notebook if you are using decision tracking (e.g. LADR). You can use the %save magic or just use additional files to create data for TDD, especially if your spike is calling a real API with sample data making the fixture data as realistic as possible.

Who works on the notebook?
This really depends on the spike, but a common format is pairing a product manager/owner with a developer to flesh out the notebooks to start with and then let the developer loose from there. The nature of notebooks makes it hard to share them for co-editing, so physical (or virtual via screen-share) pairing works best. The pair will usually present their summary and selected highlights at a weekly show-and-tell.

Spike Notebooks

Spike Notebooks provide a clean and concise way to capture the researching of new software solutions. Handled well they can dramatically improve the transparency of the work done on the spike and lead to better sprint estimates, decision making and BDD/TDD approaches.

10 years later

My previous blog post was nearly 10 years ago. At that point in time I had changed job and was embarking on a hefty daily commute; as a result I left the online world relatively untouched.

But we are back into exciting technical times and I can't stand by and let my thoughts stay un-shared.
Expect more soon.

Create UML diagrams online in seconds, no special tools needed.

yUML is a tool that uses the URL to generate a diagram on the fly.

Here is an example:

<img src="http://yuml.me/diagram/class/[Customer]<>1-orders 0..*>[Order], [Order]++*-*>[LineItem], [Order]-1>[DeliveryMethod], [Order]*-*>[Product], [Category]>->[Product], [DeliveryMethod]^[National], [DeliveryMethod]^[International]"/>

Gives you:



The technology behind it appears to be Ruby on Rails, which begs the question "will it still be useful when it gets overused?", lets hope there is a cloud waiting to host it, because this could really take off.

Fluid Requirements (Going Down The Pan)

Had this from a customer today:

Please find attached the latest version of the XXX Spec. As communicated before this document has been signed-off in-principle as it keep evolving.

Oh joy, at least it is not a fixed price contract.

BBC iPlayer finally available on Mac & Linux

BBC NEWS | Technology | BBC iPlayer now available on Mac

Go here to get it.

Bamzooki still waiting

Blogged with the Flock Browser

JavaFX = Processing + (Axis2 - i18n)/2

JavaFX = Processing + (Axis2 - i18n)/2

What type of Web services can I call from JavaFX applications?
JavaFX applications can call any REST- based Web service that returns ASCII data. If the returned data is in XML or JSON formats then they are automatically parsed for easy access. Examples of Web services that can be called from JavaFX are Yahoo Local and Flick Photo Web services.

JavaFX FAQ's | Howto and General Questions About Java FX

I think I will only pay slight attention this one until it is actually a step forward.

Blogged with the Flock Browser

The problem with measuring elapsed time on a laptop

Someone pointed out Rachota Timetracker to me the other day, probably yet another hint that I should try and do my timesheets on time. It probably came from Elijah Alcantara's comment on this digg.

My requirement is that I need a quick way to record what I am doing, but I am not always connected to the web (although I am about 90% now). A typical day is open laptop at home, slap lid down go to office, move offline to meeting room, back to office, get on train to client site, open laptop in meeting, slap lid down and head home, open and catch up in evening. The laptop gets rebooted on average twice a week.

The problem I have is that the tools for measuring time, and reacting to it, tend to go wrong when my laptop hibernates/sleeps between usage. Lightning calendar is a classic example, alarms work fine from startup to first sleep but after that the drift causes alarms to go off at the wrong time, I could use a web based calendar as that would not suffer, but I am not always connected and when I do connect it can be too late.

I was hoping that Rachota would have sorted this out, but no, sadly. I have to remember to 'relax' before slapping the lid down and running.

There are a number of possible solutions but the best one I have found in my own work as been to 'observe the system clock'. The principal is that you sample the system clock at known intervals and note the difference between the system clock delta and the interval period, assuming normal operation the minor drift that you get should average out to 0 over time (using the right kind of timer). When the computer hibernates/sleeps or you get a DST changover then you might get a difference of +1 hour for example (depending if you read localtime or UTC). That way if I start a task at 12:00 and hibernate at 12:15 for 30 minutes, when I reawaken the process and Rachota would have said 45 minutes elapsed, I know that by observing a +30 minute shift I should take 30 minutes off. If someone was messing about with the system clock then they can move it back and forth as much as they like and the technique should keep up subject to the granularity of the time interval (which will vary depending on the timing accuracy required). The system clock observer would fire off TimeShiftEvents to elapsed time listeners that would be registered and deregistered as they start and stop.

I know there are scenarios where you want to measure that off time, but you could easily make that an option for the user.

I would do the change myself but I just can't stand netbeans based projects. ;-)

Blogged with the Flock Browser

Email is a battleground, a developer minefield

The leading email apps are Outlook, Thunderbird, Mac Mail, Evolution, Lotus Notes, Gmail, Hotmail, Yahoo Mail and that's about it. In terms of market share and success there are a significant number of 'also rans', but why?

My first thought was that the main contenders dominated the market so completely that no-one bothered with the others, but then I thought back to 2003 when Thunderbird first became usable (some might say it is still not) and suddenly it gained market share. Mozilla Thunderbird proved that you can go from nothing to something, perhaps on the back of Firefox, but unlike so many other apps, users stuck with it. I have only just realised why.

A bit of background is required here. Where I work we have archives of business critical emails from certain accounts, users don't have to keep half a million emails they can access old messages using a web interface to the database of archived messages. At the moment all emails they open are downloaded as '.eml' files and so open in their desktop email client (Outlook or Thunderbird), but that method means that laptop users have to remember to remove the files as they go (which they often forget to do) because of the potentially sensitive nature of the messages. My solution was to get the web interface to display the message with the appropriate cache headers, which solved my immediate need but I realised that it would expose the system to spam (especially as some users access the system from their home networks and so cannot be proxy protected). In case the full horror is not clear, this would mean that an embedded image link in a spam email would effectively gift the spammers the bona fide email address - at best, at worst using an old IE (thinking it is in the trusted zone) we have security meltdown.

It was this that made me realise just what a fine line email clients are treading between functionality and security. The webmail systems have to inhibit the browsers natural instict to connect and show, the desktop systems have to emulate the web functionality while still limiting access. Email is a battleground, would anyone willingly enter the fray? Images, JavaScript, Applets, Flash, Ajax and application attachements all present risk in remote communications and local access.

The apps that can cut it are ones that provide just enough functionality and maintain security or provide more functionality and expose their users to 'some risk' (I would say that any risk is unnacceptable, but I don't work in their security department). Both approaches have their warfare equivalent, but what are 'acceptable losses' in this context?. There is still room for another client in the market, no-one has nailed it yet. I think Mac Mail is closest, although spammers can crash it remotely at the moment. As for my problem, I have a solution but it is probably not cost effective to implement.

Blogged with the Flock Browser

Checkout svn 1.5

Subversion 1.5 has been released. For the server/repo side the changes are:

• Merge Tracking - svn now sets a property [mergeinfo] to track what changesets/revisions resulted from a merge activity. What this means is that the 'safe' merging technique of first trunk to branch and then branch to trunk is more efficiently handled.
• Better FSFS - svn now provides the administrator with the enough flexibility to work around the limitations of various filesystems. Basically the repository files can be split up and spread around as things get large (e.g. mount a new partition), and OS file caching can be optimised.
  
• Scalability Proxy - svn allows the concept of master and slave svn instances (as in 1.4), but now all writes are directed to the master (via webdav proxy) and reads can work from the slave thus keeping things synchronised nicely. This really helps with distributed development and goes some way to providing some of the features of Git and Mercurial.

In addition on the client side:

• Sparse Checkouts - the way in which you can specify what gets checked out or updated has been made more logically complete. What this means is that you need to issue fewer, more readable commands to checkout defined subsets of a tree than before.
• Interactive Conflict Resolution - doing a 'svn up' used to then leave you hunting around for the merge to do, now it pops up with a set of options. Useful if your IDE does not have conflict resolution capabilities.
• Changelists - a way of tagging files (not directories) with a name that is used to create a set of changes that can later be used by a commit or other such command. Again this is something a good IDE will normally do for you (e.g. Mylyn).
• Relative svn:externals - a long overdue feature that finally permits the use of relative paths (e.g. ../other-project) in externals definitions. Prior to this externals had to use the full path which made usage via VPN or renamed servers of this feature impossible to manage. Now you can specify workspaces as disjoint repository trees and freely re-organise the repo without trashing all you externals definitions (as long as they were expressed well in the first place). Perhaps in the next version they will go even further and outlaw all absolute paths (replace external hosts with symbolic internal name?).
  

What else...? Bug fixes and security enhancements of course - oh and if you are still using 1.3 then you just lost your support.

Blogged with the Flock Browser

Lean build cycles with antro

Antro was released last week onto SF under GPLv3. It is a kind of visualisation tool for the performance of an ant build. It gives you colour coded drill down into your build and its files, red means long running, green means quick.



And I expected it to be just another tool that works for simple examples but never for the real thing, like so many other promising ideas. And as it was so simple to use I decided to give it a chance to shine.

Simple to use the nice part here, you just add a listener and the jar to your ant command line (or drop the jar into the ant/lib folder). Run your script and then open up the UI on the output json file. If liked the way I did not have to touch my build.xml files.

It worked on my complex build (i.e. long depends tree and heavy use of antcall etc.). And I was able to see a number of interesting 'features' of my build and then shave 10% off its cycle time. So this one gets used again.

Blogged with the Flock Browser

Another missing piece in the calendar puzzle

Google's Calendar Sync for Outlook is going to go a long way to solving the "I don't want Exchange" puzzle.

My objective for some time has been to combine the blinding convenience of Google Calendar with the working patterns of my family and work environments. I want to have a set of calendars for my job and home life that are read/writable as required. I use Thunderbird/Sunbird with the Google calendar add-on 'Provider for Google Calendar' at work and my web browser at home on the Mac. The missing part was for my work colleagues who use Outlook to be able to contribute to my job calendar, and this is the part that can now be solved.

Blogged with Flock

Why Multiple Exceptions is so hot

It seems that Monday was Exceptional arguments day - Catching multiple exceptions: good or bad?

I have to admit that I could not be bothered to follow all the replies, after the 10th stock answer and the 10th wannabe architect half baked wisdom, I stopped reading.

The reason why there are so many right and even more wrong answers is that Exception handling is contextual, and in the end the best you can ever do is achieve a 'reasonable compromise'. Exceptions represent unforeseen circumstances, or at least that was the original idea [I don't want to judge programming style, what matters is accurate 1s and 0s and the maintainability of the code that produces them], and so someone needs to sit and think about the bigger picture.

I have seen some strange programming environments over the years, as far as Exceptions go they range from:

catch (Throwable ignoreall) {}

to dropping into a full blown AI fault analysis and blame system. Both techniques were correct for their context, and have served their masters well. This is one of the reasons why writing re-usable software is so hard; you cannot predict the context of its eventual use.

My advice is to avoid writing lines of code like the above, only special circumstances demand that degree of insanity. Generally you should follow these basic guidelines:

• Never lose an Exception, always do something with it, even if that is just passing it to a piece of null code (AOP later)
• Play Ball [sometimes I pander to the audience], catch and throw anew, that way you can always participate if need be.
  
• Dots in handlers are dangerous, there is nothing worse than a NPE masking the real issue.
  
• Never throw other people's Exceptions, it can make a real mess of your dependency tree. Catch then wrap or fix.
• Don't throw when you can return, thrown business related objects make an architectural mess.
  

As far as the original article goes, any syntax that promotes good practice is beneficial, and this suggestion is one that might encourage developers to avoid unhealthy temptation. Java is a general language and so I don't really care how many different ways there are to specify what you want to do as long as the Jit compiler still has the opportunity to do a good job.

Blogged with Flock

Password Expiry is Evil?

A colleague asked me yesterday if password expiry was considered best practice, and my immediate answer was no - changing your passwords regularly is best practice, expiry is an implementation technique that forces users to follow best practice.

Personally I hate password expiry with a vengeance, I change my passwords regularly on accounts that need it and I leave other ones the same for years on end, why should I be forced into a pattern of behaviour? Being asked to advise on a security policy I always say that users should be instructed to employ all security best practice advice and be accountable for their actions and inactions. Even when the implementation of this forces users to change via a password expiry scheme then it is human nature to adopt a password scheme.

The problem is that every single scheme makes passwords more predictable than before, so that a user can go away and drink beer for a week and then come back and get in at the second attempt. Is a randomly generated password that never expires more secure that a changed password that always includes the month in it? I think it is, at least in the online sense.

Randomly generated passwords' problem is that they are generally forgettable. People write them down as a result. What that does is shift the security issue from the virtual world to the physical world, which is a mixed blessing. On the one hand staff are more familiar and well trained in the art of physical security (doors, locks, keys, etc.) and more naturally protect their passwords in that world. The other side of the coin is that, in the case of espionage, physical security is one of the most commonly compromised elements.

A side effect of having active password changing in use it that you end up with your users requiring a larger number of support requests when they do accidentally lock themselves out during a changeover. What this does is encourage the support systems to be streamlined and often forget the basic authentication required before resetting a users password. Try ringing up your support desk and pretend to be your boss!

So my verbose answer to the question is that password expiry generally does not improve security because of human nature and therefore is evil.

Blogged with Flock

Thanks Linus and Junio

I just wanted to thank those behind Git for giving me the pleasure of including the word Git in serious corporate documents. On many occasions have I wanted to use that word, but only now can I use it without the threat of reprimand.  

Git - Fast Version Control System

Blogged with Flock

Back to Flock

I have been getting more and more fed up with Firefox, or more correctly its extensions. My main issue is the rate at which it gobbles up cpu & memory, so I decided a few weeks ago to find an alternative. I am still going to use FF, mainly because I cannot work without the Live Headers, Web Developer, Open QA & standards compliance checking tools. But for my general browsing I wanted something I could have running all the time without fear of grinding my machine into submission. The main issue I have with FF extensions is that I use a laptop that never really gets time to be rebooted, so it lives in hibernation for the few hours a week when I am not developing. FF extensions don't cope with hibernation well, the naive programmers who write many of these things have not thought that the clock can shift dramatically while the app is still running.

I use a number of different browsers generally, on a number of different OS, but my target environment was a Windows XP laptop, I that meant leaving out Linux & Mac based options. So I set my criteria at must be easy on the eye, supported by most websites, secure and lightweight. I started my search with Opera which is most certainly lightweight, good on security, average visually but unfortunately it is not supported by many sites I use. Then I reluctantly tried IE7 again, which has proven to be more secure than expected, I found it worked well and was surprisingly lightweight for a MS app. But after a couple of years of using FF IE rendering is ugly, large and garish, and that was enough to discourage me.

Then I remembered flock, a FF derivative with the social extensions built in. Running this with one extension 'Adbock Plus' gives me a stable environment with all the benefits of FF without the need for risky extensions. So far it works.

Blogged with Flock

Guice Flaw

It is not really a flaw, more of an opportunity missed. I have just spent a wee while converting a project from semi-spring to guice. Anecdotally, I prefer to refer to this exercise as goosing, and I don't care what your culture currently associates with that word, Google is a global culture and the act of exacting its influence may take over the definition as far as I am concerned, big up for the living English language! Mr Knowles will hate me.

Back to the techie end...
Moving to Java 1.5/5.0 as our LCD of new java products led us to re-evaluate various existing design choices. The conclusion was that big steps need to be taken, quite simply because they can and should be taken (cost is worth it). Guice appears to be the ideal selection for such a way forward because you can always regress to a Spring, Pico or home grown framework without having to rewrite too much. But having gone through such an exercise there is one remaining doubt; which is the flaw of the title.

The doubt/flaw is that seriously used Guice requires you to transfer your 'design of linkage' to a module class, which previously was in a mixture of xml and lookup code. The people who designed Guice did the right thing by making the Binder class an interface so that you can unit test your modules, which helps, but that still leaves two issues:

1. It is hard to support build systems for multiple target environments. The modules tend towards the monolithic, and even if they are testable themselves, there are still better techniques for this kind of application extensibility, e.g. OSGI. I suggest that you use inheritance and delegation in the modules, but my issue is still that the linkage becomes a development exercise in its own right, and one that required in depth Guice knowledge to boot.
2. If you try to write a generic server style application, e.g. another web container, then you are forced to create a 'team' pattern between a Module implementation and its domain specific business classes, because they have to work together in this type of context.

I managed to resolve the build issues of multiple targets by using a 'ClasspathScanner' to discover indicator files that pointed to Module classes that constituted the application. For example the Swing app would have a jar in its classpath that had a Module indicator pointing at the swing module class to be included in the injection list, while the war build would be constructed using the modules in the war path. But I think Guice should take a step in this direction, as its own processes would be best suited to this 'discovery' phase.

I suppose what I am getting at is that the strong typing of Guice is both a pain and a pleasure. I hope Guice does for the J2EE framework world what Ada95 did for the fast jets of today, i.e. try and balance the industry concerns with the realities of development. Luckily the nature of DI is such that all these things are soluble, you just might have to learn another framework in the process. The pleasure is that it is testable, the pain is that you have to code it rather than just configure it.

Ant Custom Search Engine

I thought I would try the Google Co-op - Custom Search Engine. So I created one for Apache Ant. I included some general technology sites as well as the usual enties, let me know if you have any suggestions.

lang:java HelloWorld - Google Code Search

Google Code Search is here. Wow!

Not only can you look up to see where your open source code is being used. But you can easily police its license.

As you are developing you can be constantly searching for snippets of the code you write to see how others have used the same api.

lang:java HelloWorld - Google Code Search

Spring Simple Powerful Proven

Spring 2.0 has just been released. Ironically the same day we make a strategic decision to use Spring widely in new and existing projects.

The key benefits (unordered):

• Modular Development
  
• JSP Shortcuts
  
• Best of Breed AOP
  
• JMS Processing Support
  
• Scripting Support
  
• Tight Scheduler Integration
  
• Transaction Support
  
• Enterprise Ready
  
• Easy Configuration
  
• Annotation Based Exception Handling
  
• EJB Utilities
  
• Efficient JDBC
  

But the biggest thing is that simply making your code IoC friendly improves your design and deployment options. The enterprise integrations are a bonus after that. I just wish the beans files were as simple as pico to set up.

Free Private JVM Web Host

EATJ.com are providing free hosting account featuring a private JVM. But watch out, there is no backup, they switch off your VM every 6 hours to save resources, and they bump you off if you go away for too long.

The production account is not that expensive for a private JVM coming in at about half the going rate. But it sounds ideal for a quick and dirty test environment.

I was late for work because...

Good excuse from a colleague:

I was unable to leave my house because the front door was stuck.

1 Habit of the upper echelons of software engineers

Habit Number:

#1 Just get on with the work, stop blogy blogging about it.

Note to self - follow own advice.

Got Spam? Make Money If You Are Quick

It must come as no surprise that Spammers manipulate money markets. But until I read that article I did not realise just how much of a percentage the spammer could make.

So I did a straw poll of my last 20 or so stock spam emails and looked at the online history. Fair enough, the difference is 4-6% over the period, I am always amazed about human nature. But then I realised that the time at which I received the spam emails was usually on the upcurve, so again with my small sample I concluded that I too could make 2-4% from selected scams.

I am going to carry on watching the stock spam alerts and seeing if there is a significant pattern here. I am not planning on putting my money where my mouth is, as that would be profiting from the stupidity of spam victims. But if you were an anti-spam vigilante and you needed funding it might be a good source of income.

My First GoogleCode Hosted Project

It just so happens I was about to publish a small open source project. And what happens? Google let people know about Project Hosting on the same day. It had to be fate, so I put my SF application on hold and decided to give google a go.

My motives were really to check it out for a much bigger GWT based project which is still under wraps at the moment.

This is how it went:
1. I am already a Google account holder, so I assume that I skipped the first steps.
2. Filled in the create project form. [Which has changed in the 4 hours since I did it!!!] A look at other projects gave me an idea about the labels and I quickly realised that I had to make my description the entirety of the installation instructions. (Once you are in you can create links to other web pages for documentation)
3. Straight in no delay, you even get a subdomain set up instantly!
4. Fired up Eclipse (because Java is normally my thing) and checked out my source tree. Had to fiddle with the url a bit because my svn is a bit rusty. Got my generated access password and so I had a development environment within 5 minutes of the project submission.
5. Copied into the workspace all 3 of my project files. Made adjustments to the 'powered by' statement to link back to the GCH project location. And comitted.
6. Posted the roadmap as enhancement issues against the project.
7. Got bored because it was too easy.

Comments:
a) It is nice and simple
b) There is not much exposure at the moment, so I am still going to create a project reference on freshmeat.net
c) Commit was slow, probably because there are some large projects being uploaded.
d) It is under active development, the create project page changed in the space of a few hours (probably because it was a security risk).

I will watch the feature enhancements as they grow.

My project : hackstop

Spot the 3D difference - Vista 3D - LG3D

First look at this one which is free, java and available now...

From https://lg3d.dev.java.net/ and make sure you look at the Minority Report (Go Monkey) video in the blog3d section

Then check out this bootleg:
Internal Review Of Microsoft Windows 3D Vista @ 2007, which is not out until next year.

Try it through webstart @lg3d.

If you look carefully you can see where they are trying to catch up with Gnome too.

Rip it up and start again

Same author, same problem, better understanding, clean room implementation.
LOGBack
I wish more of the old guard of OSS projects would do this.

Rip it up and start again
I said rip it up and start again
I said rip it up and rip it up and rip it up and rip it up and rip it up and start again

[Orange Juice - Rip It Up]

And why not put it on GoogleCode Hosting while you are at it.

Microsoft = Remotely Exploitable = No Protection

To illustrate the point:





By contrast...





More than 20 unpatched advisories on Win XP alone!

Weblog Discussion: Microsoft = Remotely Exploitable = Owned

Why Open Source Projects don't practice what they preach

A lot of people use OSS resources, OSS projects want lots of people to use them, but my experience is that OSS projects don't use other OSS project resources as much as their closed source counterparts.

In my experience closed source projects tend to use twice as many open source libraries as similar open source projects. So I thought I would investigate why...

Developer Ego

This was my first thought. Historically it has taken a certain type of developer to make that step towards and open source project; and knowing a few of these developers I would expect that they would prefer to show off rather than save time and effort. So there must be an element of "not invented here".

Also when you make use of a large number of applied resources your project changes from being a development exercise into one of integration. But integration projects don't carry as much kudos as green field developments for their founders. If you have an ego then you would much prefer to be known for your development skills than your ability to integrate.

The Problem With Being Open

If your users are the inquisitive kind, and a lot of them are, then they are going to look at the choices you have made regarding the supporting services upon which your project is based. That is why so many projects use Apache commons logging, for example, because it alleviates the developer from fixing that choice in stone. But not every service can be abstracted so cleanly, and bad choices can reduce the scope in which your project can be applied. The most common one is the persistence layer where a choice such as Prevayler can prevent your project from scaling in an enterprise context. But the solution, to provide alternative options, can drain your development resources and give you an increasing number of maintainability issues.

Another issue is the rate of change of your chosen supporting resources. Say, for example, your project is based on Azereus libraries which update every few weeks. The question in your mind is which version or versions do I support? If 2 projects make conflicting choices then it gets harder to use them in conjunction (as happened with Cobertura, Hibernate & ASM). The other side of this coin is that you also don't know what versions of your own project are still in general use, and so when you make changes to your API are you creating compatibility problems for those that might re-use your software? Closed source projects don't have this issue because they are rarely used outside the sphere of knowledge of the creators.

Responsibility

End user applications need to go through some degree of systems testing. Bugs in live systems need someone to accept responsibility for fixing them. And there is always a license to uphold. Closed projects have an advantage with all of these areas of responsibility because there is usually some money to pay for extensive testing of incorporated systems, fix bugs in other people's code and investigate the legal basis of the product.

But open source projects typically do not have the luxury of a budget for supporting their re-distributed software. And more scarily, license compatibility and patent infringement are really nasty issues to have to deal with.

Feeling Exposed?

So the reason why OSS projects don't make use of many of their own kind are really related to the way in which most projects are run; a small number of personally involved developers working usually for small change handouts. The effects of this scenario subtly drive the developers to work in greater isolation, which is ironic.

Contributing to Java - What would you fix?

The article:

Sun says open-source Java possible in 'months' | InfoWorld | News | 2006-06-27 | By Jeremy Kirk, IDG News Service

...inspired me to think.
What would I contribute to an Open Source version of Java?

Assuming that Sun provide a nice test environment so that I know that I will not break the sacred compatibility of the langauge and its standard libraries, then what could you do?

These are my first thoughts:

• An interface based abstraction of the Swing widgets.
  
• Better use of models in Swing widgets, e.g. ModelBasedJLabel
  
• Back 'plumbing' of JMX and JNDI support into older APIs
  

Please comment with your own.