Tag: software factory

Tech in the 603, The Granite State Hacker

Looking into Continuous Delivery / Lean Enterprise

Jim Wilcox (The Granite State Hacker)

After a conversation with my boss I just downloaded Continuous Delivery to my Kindle reader (Windows 8 / Windows Phone app). 

On first brush, I can see it’s akin to the next phase of a process I’m somewhat familiar with.  I’ve referred to it as Software Factory in the past, and even have a label for it in this blog.  I designed and delivered a “Software Factory” solution around some of these concepts at one of the few product based companies I worked at in my career…  

“Software Factory” took Continuous Integration (CI) up a  few notches by automating not just builds, but code generation, deployment packaging, and then an early stab at what might now be called a private cloud, where we automated spinning up an instance of target test systems, deployed the fresh baked product, ran smoke tests, and (if smoke tests looked good) notified the folks in QA for more complete testing of a solid candidate.   


I’ve been a fan of Domain Specific Language (DSL) for the purpose of custom build & automation ever since. 

For that company our solution was an on-premises deployment, so it was still a matter of convincing customers to deploy the updates…  but our automation produced both full install kits as well as patches that would upgrade a running system from a known build to a target build.    


That company was preparing to take the product into the SaaS model which was very new at the time, and this would have been a part of that, as well.   

I’ve worked with TFS to integrate smoke testing and build automation, as well.  That could easily extend to this sort of thing. 

That company suffered from lack of customers, unfortunately, but it was a process I’ve been looking forward to getting back to, and expected we’d get back to in the consulting world some day.    It’s interesting that Software Factory (as Continuous Delivery) is working it’s way back to top of mind relevance through the cloud…  I’m looking forward to reading what the authors of this book have to say about it.
Tech in the 603, The Granite State Hacker

Metadata in Software Development

Jim Wilcox (The Granite State Hacker)

In modern programming, there’s a lot of cool stuff that can be done with metadata. Failing to consider it in one form or another is setting aside a major tool in any system architecture. Sadly, I see this from time to time, a project with a lot of information associated with it… typically the metadata ends up being expressed only as requirements… and is not applied any further in automation. My post on Edgewater’s blog highlights code generation as one of my favorite ways metadata can be leveraged using things like standardized requirements documentation as metadata.

Wikipedia argues that metadata is divided into two categories, “data about data types” such as XSDs and WSDLs, and “data about content” is only helpful in determining application.  According to their current article, metadata is akin to information about a container such as a box..  how big is it?  how much can it hold?   Metacontent is information about what the box contains… what is it?  how much does it weigh?  what are its storage instructions?  what are the contents of the instruction manual?  The former information is often best applied at design time.  The latter, is often more economically applied at runtime.  I would argue that it goes deeper than that, as well.  One man’s metacontent is another’s metadata. The term metadata could be used to cover:

  • raw data
  • structure/language definition (syntax) (describing how to express a schema) 
  • structure/language expression definition (schema)
  • content
  • content description/classification

Metadata in information systems has two major aspects: expression and application. Expression is everything from the media used to the language (even characters) that are used to communicate information.  Expression implies how the information can be managed and transmitted, and even how the information can be applied. Application is all about how the information is used, impacting utilization of resources requried to leverage it at the time that it is applied. 

In application development, metadata can take many forms.  It can look like lots of different things.  Here’s a list enumerating a few forms in the development world, along with pros and cons of each:

  • BLOB (Fully Custom/Encoded Expression)
    • Notes:  At the core of all other computer data expression is a specialized BLOB.  I mean, on disk, there’s no visible difference between an Excel Spreadsheet and a JPEG image… it’s just a block of bytes.  The meaning of each byte is interpreted by the application that uses it.  (Its beauty is in the eye of the beholder.) 
    • Pro’s:  Most flexible expression.  Any information that can be expressed within a computer can be encoded to live in a Binary file. 
    • Con’s:  Any “beholder” must typically be created from scratch, making it typically tightly coupled with its application.  Harder to manage in terms of media & transmission. 
  • Plain Text
    • Pro’s:  Very flexible, customizable, can be loosely coupled to the application.  Lots of options with respect to media and transmission.
    • Con’s:  Plain text can take any form, often has to be parsed using custom tools, care has to be taken to make sure plain text is extensible, human readable, and structured enough for processing.
  • XML
    • Pro’s:  Much more structured by definition, easy to extend by definition.  Many many useful tools exist to define, manage, communicate, index, and consume XML.  Very loosely tied to its application(s).
    • Con’s:  Not quite as human-readable as say, plain text might be.
  • JSON
    • Pro’s:  Extensible, structured, blurs the line between code & metadata, since JSON (JavaScript Object Notation) is evaluated by runtimes as code.  Meant for web client consumption.
    • Con’s:  not nearly as broadly supported as XML, tools are not available in as many contexts.  Not so well supported for consumption outside web clients.
  • XAML
    • Pro’s:  XML based, and like JSON, is an object initialization notation.   It is more widely known for describing UI elements in WPF and Silverlight.  Directly compiles to code.
    • Con’s:  Requires WPF or Silverlight runtimes to use, not well supported beyond client-side rich apps, media may be typically described as “embedded in binary code”.
  • Embedded in Programming Languages
    • Pro’s:  typically compiled into code, metadata can be expressed as code litterals or code attribution.  This means the data is almost fundamentally instantly available to the running application that consumes it.  Done well, especially with code attribution, this tends to provide meaningful human-readable information to a programmer (typically more so than comments) and a runtime processor at the same time.
    • Con’s:  embedded metadata must be processed at runtime, every time.  In cases where there are large amounts of data involved, done wrong, this can be time consuming.  Typcially forces metadata content management into source code control, which arguably may not be the best way to manage it. (source control of metadata is a “Pro”, but putting it in something like TFS can make it hard to reach.) Litterals and attribution can tie metadata to the code that consumes it.
  • Excel Spreadsheets
    • Pro’s:  typically easily read & shared, easy to manage & communicate as a file. 
    • Con’s:  not so easy to collaborate on, typically concurrent content managers must merge changes.  Automation can be klunky.
  • Database Tables
    • Pro’s:  Easy medium for a developer to work with
    • Con’s: Not so easy a medium for anyone else, hard to manage in source control, requires custom tools for end-users to work with if they need to.
  • SharePoint Lists
    • Pro’s:  Easy medium for anyone to use / collaborate, SharePoint provides content management tools.  Data is easily decoupled, accessible via web (and web services), and exportable to files such as Excel spreadsheets.
    • Con’s:  Hard to keep metadata sync’d with source code control.  (If this is a requirement, the workaround involves exporting documents and checking them in.)

Metadata application, in programming terms, can be accomplished in several different places.  It can be applied at:

  • Design Time
    • Common application:  code generation based on metadata or metacontent information.
  • Run Time
    • Common application:  runtime behavior/presentation modification and/or extension based on metacontent information.  This could include interpreted language processors. One could argue that .NET Common Intermediate Language (CIL) is metadata for that reason, since a runtime interprets it and executes native code from it.
  • Test Time
    • Common application:  baseline test data, test configuration.  (In turn, these can be applied at “design time” such as generating unit test code, or runtime, such as gathering baseline test data to compare to results, or determining executability and/or parameters for tests.)
  • Deploy Time
    • Common application:  Deployment targets/configuration based on build content metacontent.
Code generation is one of the most compelling uses for metadata in information systems, but it’s certainly not the only use. One of my other favorite topics, SharePoint, highlights this by making metacontent a cornerstone of content and document management, classification, reporting, and search indexing. Throwing a document in a plain old file share provides you with a little metadata; the folder names it its file path typically has meaning, as well as the dates, times, and even the permissions applied to it. Throwing the same document in SharePoint provides a ton more information about the document content by default, and typically indexes it, making search a far more powerful tool.
Tech in the 603, The Granite State Hacker

Anonymous Form Submission to Form Library with InfoPath in MOSS

Jim Wilcox (The Granite State Hacker)

Here’s a bit of a trick I ran across while helping to develop some MOSS2007 solutions.

I needed to configure InfoPath so that it could submit documents to a site that the submitter would not be able to access. In SharePoint, this is not directly possible.

A common work-around is to set up incoming email for the target list, and submit by email to that. Unfortunately, my client is part-way through a Notes to Exchange migration, so this wasn’t practical in the given time frame.

The solution… create two sites, one that is accessible to the submitter, and the other that is not. On the accessible site, create a new, “hidden” list that the user can submit to. Add an event receiver to that list, such that whenever a new item is added, the item is moved to the real intended target using elevated privileges.

Using VSeWSS extensions, create a List Definition project that has something like this in the ItemEventReciever.cs file:

 

using System;
using System.Security.Permissions;
using System.Runtime.InteropServices;
using Microsoft.SharePoint;
using Microsoft.SharePoint.Security;
using Microsoft.SharePoint.Utilities;
using VSeWSS;

namespace AutoForwardingFormLibrary
{
    [CLSCompliant(false)]
    [TargetList("38aea319-af78-4489-9059-d124c68bf9fe")]
    [Guid("9d0f139b-d9ed-4b6c-b0ba-2353cb3bad85")]
    public class AutoForwardingFormLibraryItemEventReceiver : SPItemEventReceiver
    {
        private SPListItem addedItem = null;

        /// 
        /// Initializes a new instance of the Microsoft.SharePoint.SPItemEventReceiver class.
        /// 
        public AutoForwardingFormLibraryItemEventReceiver()
        {
        }

        /// 
        /// Asynchronous after event that occurs after a new item has been added to its containing object.
        /// 
        /// 
        /// A Microsoft.SharePoint.SPItemEventProperties object that represents properties of the event handler.
        /// 
        public override void ItemAdded(SPItemEventProperties properties)
        {
            addedItem = properties.ListItem;
            if (addedItem == null)
            {
                throw new ArgumentNullException("properties.ListItem");
            }
            SPSecurity.CodeToRunElevated moveItem = moveItem_Elevated;
            SPSecurity.RunWithElevatedPrivileges(moveItem_Elevated);
        }

        private void moveItem_Elevated()
        {
            addedItem.CopyTo(SPUrlUtility.CombineUrl("http://targetserver/sites/jimw/docs/formlibrary", addedItem.Name));
            addedItem.Delete();
        }
    }
}


Tech in the 603, The Granite State Hacker

Application Platform Infrastructure Optimization

Jim Wilcox (The Granite State Hacker)

In doing some research for a client on workflow in SharePoint, I came across this interesting article about the differences between BizTalk 2006 and the .NET Workflow Foundation (WF).

The article itself was worth the read for its main point, but I was also interested in Microsoft’s Application Platform Infrastructure Optimization (“APIO”) model.

The “dynamic” level of the APIO model describes the kind of system that I believe the .NET platform has been aiming at since 3.0.

I’ve been eyeing the tools… between MS’s initiatives, my co-workers’ project abstracts, and the types of work that’s coming down the pike in consulting. From the timing of MS’s releases, and the feature sets thereof, I should have known that the webinars they’ve released on the topic have been around for just over a year.

This also plays into Microsoft Oslo. I have suspected that Windows Workflow Foundation, or some derivative thereof, is at the heart of the modeling paradigm that Oslo is based on.

All this stuff feeds into a hypothesis I’ve mentioned before that I call “metaware”, a metadata layer on top of software. I think it’s a different shade of good old CASE… because, as we all know… “CASE is dead… Long live CASE!”

Tech in the 603, The Granite State Hacker

Art & Science

Jim Wilcox (The Granite State Hacker)

Fire & Ice… Day & Night…

This question, Art vs. Science, has come up a million times in software development circles. Reading Paul Johnson’s (Paul’s Pontifications) blog post, in conjunction with a discussion in the Tech Mill at Edgewater, (thanks, Jason!) I have come to see that art and science are not as opposite as I once viewed them to be.

What hit me was that Paul makes the statement that there’s no process to implementing software. I still disagree. There are many processes.

The number of processes that an implementer can choose from to write his/her code is often vast, and depends on the problem set. A problem set includes many things, including requirements, tools, target platform, development platform, existing code, and even the implementer’s mood and frame of mind. That is what makes implementing code, like painting, or creating a recipe, an art.

Within a common implementation problem set, there can be a large number of processes which can be applied to derive valid solutions. In fact, there are so many, that some distinct processes may actually render the very same code. So, to be more clear, it’s not that there’s no process… it’s that there’s no single valid process.

Knowing that there’s no one single valid process doesn’t mean that we can’t pick a needle from the haystack… if the process produces a solution within the problem set, it’s good.

Now consider what happens when you start to narrow a problem set. There’s lots of things you can do. Frameworks, platforms, clear-specific requirements, best practices, coding standards, well structured architectures… these things are all factors that limit the problem set. By narrowing a problem set, you narrow the number of valid processes. By narrowing the number of valid processes that a developer can choose from, lots of interesting things start to happen. You achieve more predictable results, and are more likely to achieve repeatable schedules… and you reduce overall project risk.

This is what’s so interesting about contemporary trends in software development, such as Ruby on Rails… use of these tools narrows problem sets that developers face. This means the implementer can spend less time figuring out where the blanks are, and more time filling them.

Now let’s take this further. What happens when you reduce the problem set dramatically…? Take a single, relatively well known problem, on a very specific platform, using a very small set of unambiguous expressions. You get a very tightly defined process. By doing this, you wring the art out of creating something, to the point where it becomes machinable. The process becomes realized as a factory.

So to answer the question… Art or Science?

It’s a trick question… art and science are not exclusive opposites. Art is about freedom to choose your creative process. Science is about knowing what processes are available, and the pros and cons of each. So programming, like all creative activities, is usually art (except in single-processed cases), and usually science (except in cases of serendipity and true miracles).

Paul’s Pontifications: An Under-Appreciated Fact: We Don’t Know How We Program