When I first developed AgileSites, I tried to address a common issue: the endless loop of fix-the-html-when-the-agency-updates it. I solved the problem but I was unaware of the scale such a problem can impact real-word projects.
What is the problem with JSP?
In the ideal world in the mind of people who designed JSP, HTML designer are so smart to be able to directly edit the JSP when they design the HTML. Better: HTML designer should be so smart to figure out exactly which template they should edit (knowing the content model perfectly) and change it directly in the CMS, then publish it and the site is updates
In the real world, HTML designers have no clue how the CMS work, and no idea of the underlying content model. Better: they don't even want to know: their job is complete once they delivered the mockup of the HTML. It is the JSP coders job to implement it in the CMS
Also in the ideal world of Web Agencies, the HTML is done only once, and it is perfect. It completely satisfies the requirements, it works with all the browsers and the mobile devices in the world, so it can be handed over only once to JSP developers for implementation in the CMS.
In the real world, the HTML is a never finished work. It is first delivered incomplete, then it is updated more or less one thousand of times before it is complete. In the meanwhile the customer changes his mind on the average 2 times a day, and the HTML designers keep updating the HTML mockup.
In the middle, JSP developers go crazy cutting and pasting and diffing the changes made to the HTML, they struggle fixing the JSP to keep up with HTML changes, also because there is a lot of rendering logic embedded in the HTML.
What is the AgileSites solution?
Because I had a first-hand experience of this nightmare, I designed AgileSites in order to keep the HTML totally separated from the Java code, allowing for totally independent editing of Java and HTML.
The basic idea is pretty simple: you keep the HTML in its original form. You only ask HTML designers to clearly mark where the content should go using normal HTML conventions (usually ids and classes).
When the HTML is delivered, Java coders simply write Java code that will load the HTML in its original form, cut it in smaller pieces (and this cut work is done in code, not manually) then replace the HTML with the content taken from the HTML. That is all. This is what AgileSites allows you to do. This reason alone is enough to use it.
The great benefit is that now HTML editors can change the HTML at their hearth content. As long as they keep the ids they defined in the first place (and usually this is something they do because an id is a convenient placeholder for something they need to keep track of) the HTML can be completely restructured, the look can change completely.
All of this without impacting the Java code that is already written. Java code must be changed only when new content is needed, not when the look of old content changes.
Some numbers
When I first designed this system, I was unaware of the scale this method can impact the development process of a large site. In a large project, done with just Sites and JSP, the HTML was changing all the time. I would say that is normal
In that project, there were just 3 HTML developers, but for implementing that in the CMS 5 JSP coders were needed, busy all the time changing the JSP and writing tons of code to keep up with the pace of the HTML developers. The time to implement a change was one week at the minimum for simple changes, to became 2 to 3 weeks when the HTML changes were more complex.
I am now working in a much larger project, very rich in HTML and Javascript coding. Luckily for them, they adopted AgileSites. Numbers here are very different: there are at least 50 HTML/Javascript developer producing HTML changes at a very high rate. Applying the numbers I saw in the preceding project for keeping the JSP updated you would need maybe 70 JSP coders!
But thanks the AgileSites approach, only TWO Java developer (yes TWO) are required to do some occasional change, that is usually done in a matter of minutes!
Thanks to the feedback of our users we improved the quality of RC1.
The demo for the 11.8 has now all the tests working, and the csdt export works cleanly.
Improvements in the build
I merged some improvements in the build that were already developed in a separate branch.
You can specify the site to export directly with the "!Site" syntax, so you now export only one site with:
wcs-dt export #Demo-11.8 !Demo
The installation commands has been simplified.
You install everything with the single wcs-setup while Sites is offline, import a site then and deploy your code changes with the single wcs-deploy command, that will eventually automatically import some elements.
There is now a wcs-hello command that checks if Sites is running and its java version (and checks if it matches with the java version running AgileSites to avoid discrepancies).
There are a number of other internal things tidied but no other user visibile changes.
After 2 projects done by my team with AgileSites and a lot of feedback from everywhere in the world, we are finally releasing AgileSites 1.8-RC1.
This is the first release coming from the London-based Sciabarra.com ltd officially supporting it commercially, and I am the CTO. The project is still Open Source and Apache Licensed, and it will be stay free for all users forever, including commercial ones. However since some larger organizations were looking for a commercial entity supporting it, here is it.
The biggest thing missing for a full release is the improved documentation, that I am writing. However the current documentation is still largely valid, and this blog post summarized the new things, so for a few days I hope you can live with.
So without any further ado, here is what is new.
Ready for Sites 11.8
The latest release of WebCenter Sites is 11.1.1.8.0, and AgileSites is ready for it
We introduced support for Web Center Sites 11.1.1.8.0 and retained 11.1.1.6.1 .
We now supports multi-devices and multi-web root with the same api as before. A new mechanism to write Elements that render different for each device has been introduced.
Improved TOOLS
We introduced an automatic configurator for quick installation. You are no more required to edit the configuration file, a GUI will pop up and detect automatically the correct parameters.
The Eclipse plugin previously generated 3 different projects and required Scala-IDE (althoug the code was all Java). This is no more: there is only one project for your app, and a java-only IDE is all that is needed. Basically any flavour of Eclispse is enough. IntelliJ idea can be used also, and only one project needs to be imported (the one where you do your development work).
CSDT interface now supports multiple workspaces. In the command line you can use this sintax:
wcs-dt import #MySite-11.8
to import from the "MySite-11.8" workspace.
Improved API
The API has been reorganized and extended, supporting easily also dependencies and recommendation (you don't need any more to use tags for this). Part of the API is now in the core, defining a set of "key" interfaces, while the implementation is an "add-on". This distinction because the API will have a new pluggable implementation, accessing to the content with REST services.
A very nice addition to the Picker templating engine is a "moustache" syntax for extracting attributes. So you do not need to do any more this for a simple attribute:
html.replace("#id", e.getString("Title"))"
but just write:
<div>{{Title}}</div>
and the Picker will extract directly the attribute from the current asset. This is part of the newly introduced Content interface. The Asset, the Env and a custom Model are all contents that the Picker can use.
We removed some oddities from the naming convention, so everything now it is uniform. In the code now the name of the element you call has the same name as the underlying class. You are recommended to give a consistent prefix to all types and subtypes (and ideally that should be the site name), separated by "_". Any prefix is removed from your code.
Under the hood
Lot of work has been done under the hood.
The most important is that we revamped the classloader. Now it is multi-jar (it was single jar previously). The big advantage is that you can now drop multiple jar, having them hot reloaded.
This opens a lot of possibilities, including full multi-project and plugin support (that we are going to leverage in the next release). Also it is more efficient because the poll is done only once in a while (currently once every second) allowing to use it even if your site had thousands of calls per second.
We worked hard to refine details to clean many stupid warning and errors and . For example, now in the core there is no more an "agilesites-core.jar" but an "agilesites-core_11.1.1.8.0_1.8.0.jar" with the full version number (just in case it is not clear).
Another stupid but not easy task was to remove warnings when you started catalog mover (now integrated in the shell too).
Java Server Pages are at the hearth of Fatwire and Sites development. The core activity in CMS development is extracting content created within the system to render it in a web page. Normally the web designer provides the HTML mockup, then the developer write code to replace key parts of the mockup with the actual content stored in the CMS.
Writing JSP templates is the main task of the programmers in charge of implementing the mockup in the CMS. Sites programmers may also choose to code XML templates, using the less powerful XML based proprietary language of Fatwire/Sites.
There is another option today: pure Java controllers with pure HTML views, using the AgileSites framework; it can significantly simplify the work of implementing the mockup, as described below. However, let us discuss the available options in order.
You may have significant problems with JSPs, because it is a technology not designed to implement business logic. Indeed, JSP is only a presentation layer, the View Tier of the ubiquitous MVC pattern. The pattern actually assume a controller layer. Unfortunately, a controller is not available out of the box in Sites, only the view. Furthermore, standard tags actually are verbose and requires the developer to perform all the steps to extract the model inside the view layer (a job that the programmer should delegate to a controller layer).
So in Sites often we end up with business logic embedded in the presentation layer (the view). Sometimes, it is simple business logic (just read the content). However, it can became pretty complex especially when you need to perform integrations. Long and detailed JSP, containing much more java code than HTML , are frequent.
Why you should not use JSP for complex logic
Writing java-filled JSPs are problematic because:
They are not easy to understand since they mix different levels of logic
The Java usable in JSP is limited (for example in a JSP you cannot define a class ) so you end up coding with a crippled Java
Sites API creates a global environment not really (containing only of strings and lists), that is not a very powerful environment to express complex business logic.
As a result of the considerations before, I consider the JSP only template technology of Sites a big drawback that hinder productivity and in many cases actually put projects at a risks.
It is not an exaggeration. Why?
Because developers working under pressure rarely have the discipline of follow healthy coding styles. Many projects ends up in a collection of large, hard to read and understand, complex spaghetti JSPs. Those JSPs use crippled Java so they are repetitive and often filled of copy-and-paste code. Finally they implements business logic that should really go in a controller layer.
So what we can do?
Proper separation of Presentation from Content
What we need in my opinion is the following:
A way to really separate the controller logic from the rendering logic (MVC)
The ability to keep the mockup in his original, HTML only format.
A nice, easy way to place content in the original HTML mockup
A way to cut a snippet of the complete mockup to use it in a template that renders a pagelet.
You can do it all with AgileSites. The framework (that is completely open source, free and usable with no license charges in commercial projects) satisfies those requirements by design.
Let's see how.
First, AgileSites lets you to code in Java everything. The details of its magic are in this post:
The simplest possible implementation of a template is then a Java class like this:
publicclassHelloextends Element {
@Override public String apply(Env e){
return "<h1>Hello, world</h1>";
}
}
Considering only this sample, AgileSites provided a solution for writing business logic using full Java (you can create classes, use inheritance, share objects and so on)
However, it looks like we created a problem that JSP were supposed to solve: producing HTML in pure java is cumbersome.
Actually, this java class is only part of what the framework provide: indeed, it provides out of the box the Picker, that addresses the HTML generation problem in a good way.
In AgileSites, the programmer uses the HTML mockup provided by the Web Designer, leaving it in its original form. Here is the code you can use to render the mockup with the picker:
@Override public String apply(Env e){
Picker html = Picker.load("/blueprint/template.html");
return html.outerHtml();
}
This code will load your mockup and emit it in output. We reached goal 2 (keep the original html), and we are close to reach the goal number 1 (separation of html from code), in an MVC way.
Of course, that is interesting but it is not enough. For example we may want a template that is only a snippet of the mockup. What we can do? Usually in JSP template development, we do exactly this: we cut snippets of the mockup and place it in the JSP. A practice that makes updating of the template a real pain. In AgileSites you do not need to do this. Instead, you can use this code:
Picker html = Picker.load("/blueprint/template.html","#content") ;
return html.html();
It is enough to cut the snippet we need. The Picker will “pick” (hence its name) the snippet identified by the id “content”. The syntax for selecting the snippet we want, is the same used for CSS and the ubiquitous jQuery library. The CSS selection syntax is so well know that should not create any problem to Java developer of any level with some exposure to the web. Note that we used the “html” method, that should probably be named “innerHtml()” since it returns the inner html of the block selected with “#content”. The the outerHtml() we used before returns the html including the node you selected.
Now things should became easier to understand. We also reached goal number 4, cutting snippets from the original HTML. However, we still need to place the actual content in the HTML. Here is where the real power of the Picker lies: it lets you to replace parts of the original mockup with the actual content.
The picker, combined with the API to access attributes described in the previous post, creates a very powerful tool to implement easily templates keeping the original mockup and let us to reach all the goals we discussed before.
This code:
html.replace("#title", a.editString("Title"));
combined with this html code in the mockup:
<h1 id="title">Welcome</h1>
lets you to replace html in the mockup with the content stored in an attribute.
But wait: there is more
Actually, the picker is much more powerful that just this. You can replace attributes, delete nodes, use complex selectors, append and prepend to existing html and so on.
You can get an idea of what is possible to do checking this example:
Fatwire (now Sites) API was defined around 1995 and was pretty cool... for the hostoric period when it was developed . At that time XML was pretty new and everyone was wondering which purposes it could have. The obvious idea was to use it as an extended programming language syntax.
Originally Fatwire API was based on a executable XML format, and every function is a tag. The whole idea was later moved in the JSP world with custom tag libraries. So what you have today is a set of JSP tags to do everything you need, including loops, conditions and variable management.
The problem is that the power of the tag based programming is very low compared to what you can do in Java, so you end up using a crippled Java managing a pretty primitive and verbose environment.
A great example of this problem is the code needed to retrieve an attribute. You have to do all of this:
Now finally you get the attribute value in the single variable "title" (that you still need to access as ics.GetVar("title"))
What this code does? Simple: it retrieves a java object (what is today called a Pojo) that is a value object holding part of the current asset. Then the code extracts from the Pojo the requested attribute as an IList.
An IList is the internal datastructure used by Fatwire (that is actually a cached result set, and it is more similar to a table than a list). Finally, you can extract the field value from the first row of the IList.
The code is so long and verbose because it uses the tag syntax (definitely not a good programming language syntax) and also exposes details (the IList) that should be encapsulated in some object. Ideally, the Pojo that encapusale the asset should be also have the attributes internally, not to be retrieved separately.
How all of this can be better?
Enter AgileSites, the open source Framework for Sites and Fatwire for Agile development. It offers a way simpler API to do the same operations: extracting an attribute.
Because AgileSites provides a way to code your template in Java (as explained in this post) , while keeping the full ability to invoke the Fatwire API, it was not too hard to implement wrapper cpde that makes this very frequent occurrence much easier.
In AgileSites you write:
Asset a = e.getAsset();
String title = a.getString("Title");
The "e" variable holds the current enviroment (actually it wraps the ICS object with additional features) . Since you can assume that the variable c and cid refers to the current asset, the whole
Now the a variable holds the current asset and provide a number of methods to extract various informations. Since extracting attributes is way the commonest occurrence, a method getString is provided. The equivalent of
How it works? First, all the underlying ILists are created lazily when needed with an unique name to avoid clashes. Then since the lists used to extract attributes uses always a field value, it is not necessary to specify that parameter. As a result, all you need to do is to specify is the name of the attribute and in which data type you want to convert it.
Actually there is much more. The API allows to access multiple fields, create links and even define slots and so on. But I will talk of this in future posts. Stay tuned.
In this post I want to explain the underlying ideas behind AgileSites.
The goal is not only showing what it does but also why and how it performs its magic.
In theory you may also apply the ideas in this article without using AgileSites itself, although I believe the simplest path is just download and use it if you like the approach. There are no strings attached: the framework is Open Source and Apache Licensed (so you can use it in your commercial project without having to release any code).
Simplifying Sites Coding
Everything starts when a Java developer looks at this piece of JSP code (the typical code you have to deal with when you code for Fatwire/Sites)
<assetset:setasset name="as"
type='<%= ics.GetVar("c")%>'
id='<%= ics.GetVar("c")%>'/>
<assetset:getattributevalues
name="as"
listvarname="ls"
attribute="Title"/>
<ics:listget
listname="ls"
fieldname="value"
output="Title">
<%= ics.GetVar("Title") %>
All of this coding is needed just to read an attribute. Code is so verbose because it uses JSP tag libraries. Tag libraries are not really designed to implement a complex business logic, only simple rendering logic in a form readable (and editable) by people who cannot (or does not want) code (for example HTML designers).
As many people point out, XML (and thus HTML) is not a programming language, and if it were, it would be verbose and unreadable. This code is a typical example of a simple task becoming complex because the tool is meant for a different purpose. What this code does, after all, is just retrieving an asset and then extracting an attribute from it.
A Java/OOP developer would immediately say: the intermediate IList used is a technical detail that can be hidden inside some object, and many parameters (like "c" and "cid") can be considered implicit, so they do not need to be expressed.
So if we were using Java, you would define a couple of classes and the code could become as simple as:
e.getAsset().getString("Title")
Indeed this is how attribure retrieving looks in AgileSites, and this simplicity is one of the (many) benefits it provides. The problem here is to understand how it reaches this goal.
Switching back to Java
The first idea to simplify Sites coding is to use a library. In plain Fatwire/Sites you can do something like this:
<ics:callelement name="Lib/ReadAttribute">
<ics:argument name="attribute" value="Title">
<ics:argument name="source" value="as">
</ics:callelement>
Indeed many people uses a library of reusable elements to automate attribute retrieval. The result however is not much more readable than before, only slightly shorter; but elements (because they are not really meant to be an OOP construct) have problems like changing the enviromnent with sometimes unwanted side effects. There are thus drawbacks when using a library of elements.
A plain Java class as a library is much better. So why not use it? Indeed AgileSites defines a couple of classes similar (but more complex than) to those:
class Asset {
String getString(String attribute)
}
class Env {
Asset getAsset()
}
Internally the Env.getAsset() actually performs the assetset:getasset and stores the result in the class instance (code is optimized to load the assetset object only once). The Asset.getString() actually uses the assetset:getattributevalues to retrieve the attribute (code is optimized to load attributes lazily).
All of this is not a great magic for a Java coder. The difficult part is how to write Java code in the JSP-only world of Fatwire/Sites environment, invoking using Java tje API available only as JSP tags...
Java coding in a JSP world
Writing code in Java is not uncommon in the Sites development world. The real problem is that the process is not dynamic and productive enough. In J2EE world if you add a Java class to the webapp, you need to redeploy it to be able to see changes in Java code. Because all of this is pretty time consuming, Fatwire/Sites developers does not do this very often, and instead just rely on JSP (stuffing a lot of java code in it).
AgileSites strives to be agile, so instead of asking the developer to redeploy the application for each change, it does its magic with a core library that can load dinamically a jar built continuosly without restarting the application server.
It is important to note that this ability is not new at all in the Java world. Applets has been working this way for ages: they load in the browser a Java application packages in a jar from the net, and a browser restart is not needed. Indeed, since the ability to load a jar is built into Java from version 1.0, AgileSites uses the Applet class loader to load such a jar.
So your code can be just plain Java. Templating is done with the Picker keeping the view in pure HTM; I will cover this feature in another post. You code your business logic in full, uncrippled, clean Java. Then your code is compiled on the fly and packaged in a jar and it is available for execution putting the shell in continuos compilation mode.
When you invoke a template, the jar is reloaded (only if it is changed) and then your code can be executed. This way your java coding is as dynamic as JSP coding but you do not have to use the crippled java offered by JSP.
The whole trick is explained in this Architecture diagram:
Invoking tags from Java
The next problem is writing code that can actually use JSP tags. Fatwire provides a Java API for asset manipulation but many functions are only provided as a JSP tag. Actually JSP tags in Fatwire/Sites are only a proxy for invoking the internal Sites API (really the one designed for use with the XML tags) exposed through the ICS facade. However invoking a tag with ICS is pretty awkward. This is why AgileSites provides a layer for simplifying access to tags.
When AgileSites is built, it will inspect the Tag Library Definition of Fatwire/Sites to build a wrapper library to access tags from java. Using this wrapper library there is a direct mapping from a tag to a java call, than can be show with a simple example. A tag invocation like:
Using this library for example the retrieval of an attribute can be implemented with this following code (equivalent to the assetset:getattributevalues but encapsulated in a clean java class).
public String getString(String attribute) {
return e.getString(at(attribute), "value");
}
private String at(String attribute) {
String attrList = attribute.toUpperCase();
if (i.GetList(attrList) == null) {
String attrType = e.getConfig()
.getAttributeType(ics.GetVar("c"));
AssetsetTag.getattributevalues().name(as)
.attribute(attribute)
.listvarname(attrList)
.typename(attrType).run(i);
}
return attrList;
}
If you read this sample you should be able now to understand fully how is possible that the complex JSP coding of the first example can became the simpler Java of the second example.
However AgilesSites offers more: for example the pure-html templating . Stay tuned for the next post if you are interested.
After one year of work, and after successfully using the framework on real world customer, I am happy to announce the release 1.0.0 beta1 of AgileSites.
AgileSites is a revolutionary framework (for Fatwire and Sites) that brings a lot of features, mostly oriented to allowing modern Agile development.
I seriously believe that Fatwire and Sites development will never be the same.
Most important the framework is Open Source under the commercial friendly Apache License 2.0
But instead of talking of it, watch the video, then consult documentation in its website: www.agilesites.org
