Tuesday, December 20, 2016

Shopping for Frameworks

There is nothing more nerve wracking in a software project than choosing which horse to back, which basket to put your eggs in, and which metaphors to mix. Shopping for frameworks (and it truly is shopping, you are going to spend money) is much like regular shopping. If you get in and out of the store in 10 minutes you've probably not thought through how that jacket is going to match your wardrobe and if you got something cheap you'd better be prepared to stitch the buttons back on. So with no further ado I present my humble advice on the titular topic.

Alex's Framework Wisdom

  1. There is no free lunch. You will pay for free frameworks with ongoing development cost working around their oddities and/or contributing upstream. That is to say less CapEx means more OpEx. In the case of subscription licences it's purely OpEx as salaries or as licenses, either way you'll pay.
  2. The quality of the vendor is as important as the framework. You are piling your developer hours (your money) on top of these frameworks. The vendor has the ability to kill support at any time and transmute your beautiful codebase into a pile of technical debt with the snap of their fingers. If that worries you, get a contract that stipulates they won't.
  3. New isn't automatically better. New languages, new toolchains, new companies. On the surface these may be exciting and they may give you a market edge in some cases. Each one of them is however a double edged sword. These are a lot of companies out there using tried and true technologies because they work and they are known quantities. You should be one of these companies unless there is a pressing market reason that you cannot be. Using new things will make it harder to hire staff. The staff you do hire will create technical debt as they learn what works and what doesn't.
  4. Invest in training. When you choose a framework get your development team together to learn about it. It is crucial that this happens as a group. Received wisdom does not work well in the software industry. If all your team learns about something at once it will create homogeneity and accepted best practices. Handing a rulebook to developers will not work well they'll either ignore it or they'll all make disparate readings of it.

How to avoid choosing poorly

Congratulations you think you've found just the right combination of lego pieces to solve your business case. Not so fast! Before committing you (and by this I mean not you, I mean your staff) need to take it for a test drive.
  • Apply at minimum your own standards. What does the framework's source look like? Is it public? If it's public review it. Makes sure it would pass the standards you set for your own code. If possible try fixing an open bug and getting it accepted upstream. If the framework is closed source you are especially dependent on the vendor. Look at the release notes and review the bug tracker. Make sure things are dealt with in a timely fashion. 
  • Consider the vendor's bus factor (how many of their team could get hit by a bus without impacting their ability to deliver). If it's a small number take this in mind. Will their business model keep them around at least as long as your product?
  • Check out the debugger. Write something with a bug in it and hand it to a second developer. Their job is to document how the debugging tooling works while they find the bug. Did it run right out of the box or did they have to spend an hour setting up an environment? Does it hit every breakpoint? Are values presented in a useful way? Can it attach to a running process? Plus any other metrics your team might find useful here.
  • Eyeball the UI. If it has a UI component take it to your UI and branding team. The default themes will not satisfy your business requirements. Get the UI team to specify the changes that will be required and then attempt to implement them. If this is difficult start worrying.
  • Inspect the toolchain. Does it integrate with your CI setup? Can you make repeatable builds of your code over time. If not what would you have to do to internalize any build dependencies? Does it build in 1 second or 1 hour?
  • Figure out how to test the end product. Does the framework have interfaces to test tools? Can you write a unit test, a functional test, and an integration test? Are the tests stable? If the framework has a UI component can you automate the UI testing or will you be paying people to click buttons?
  • Consider reuse. Is the code you write in the framework coupled to it? Will you be able to take code from it to other places in your business? Will you be able to bring code into it?
These are just some high level acid test questions you should be asking yourself when you are looking at a framework. Please take into consideration any other factors important to your staff and your business case.

Case study; Death by 1000 cuts (or Appcelerator Titanium)

The small issues hurt. It is not the big problems that make you regret your choices. It is the day to day grind. The subtle bug that no one suspected and no one documented. It is the lost hours in every week that your staff could have been doing something that generates revenue. See wisdom #1. Trivial hindrances also create morale issues. Monotonous and frustrating work will demoralize a group of people who typically pride themselves on doing new, interesting, and challenging things. Nothing is less rewarding that realizing you lost your day to a missing semicolon.

Below are replies to some issues I have personally raised against a particularly hodgepodge framework known as Appcelerator Titanium. They are prime examples of the kind of bugs that will decimate your productivity should they occur with regularity. Also note these aren't even bugs with the framework these are bugs with the associated toolchain, remember to investigate the whole package you are buying into.


It's a little complicated. The available Titanium SDK releases depends on if you're using ti sdk vs appc ti sdk. appc ti sdk shows the latest and greatest SDKs. ti sdk shows the latest release before the switch from the Titanium CLI to the Appcelerator CLI.

The list of releases for the Titanium CLI is no longer automatically updated, but rather it's a manual process. Because we haven't determined which past releases should be available to ti sdk, we simply haven't updated the release list.

However, the list of releases that appc ti sdk reference is automatically updated and current, so I advise you just use appc ti sdk for now.

Time lost: 1 hour


Smart quotes are not valid in XML. I wonder what the XML parser library is doing. I wouldn't be surprised if the platform was being parsed as "“iphone”". I don't think we should naively replace all smart quotes with ascii quotes and properly scrubbing the smart quotes is more effort than it's worth. In other words, I don't think we're going to improve this.

Time lost: Half day

Trivial issues like the ones above and dozens more will slow your development process and make your project management unpredictable. This has a real fiscal impact. If a developer has a high chance of running into a framework bug or a hidden issue they will not be able to provide accurate estimates of effort to management. Without accurate estimates you will either miss release dates and stress your employees trying to make back time or have long release cycles and lose agility. Neither of these is going to help you make money.


You never know everything about the ship you are getting on board. With that said a few sanity checks may save you months of rowing back to shore or having to listen to an 8 piece band play you out. Spend the time up front to get an idea of what is going on, trust the people who will be doing the day to day work, and remember nothing is free.

1 comment: