Discussing CSDT and FatStart with the architect

Val Vakar (CSDT Architect/Lead Engineer) was so kind to comment my "review" (let's call it a review - it is not) of CSDT. He raised a number of interesting point that I am going to discuss here.

I am going to compare actually CSDT with my own FatStart/ElementDeployer. Of course I am biased here since FatStart is my brainchild. However, I would be happy to drop it and use CSDT instead if it was addressing all my development concerns. Since it does not, yet, I am going to keep it and instead I ask Val to improve CSDT to the point I can drop FatStart.

I have been looking for a valid Fatwire development environment for ages, and I do not count CSE as a development environment! At the moment I am still using  the development environment  I built by myself.

Response to Val comment

Let's see what Val says:

Ciao Fatcoders!

Pleased to meet you, Val.

Just a couple of comments on your CSDT review:
1. You’re right that the fundamental difference between CSDT and Fatclipse is that CSDT works locally whereas Fatclipse works with a central server. This was a deliberate decision to bring FatWire development in line with normal J2EE practices where there’s distinct separation between development and deployment so that developers never deploy code directly from their system (even to test environments). For example, consider deploying a JSP to a remote WebLogic or Tomcat instance (and no folder sharing) – see what I mean? This approach is necessary to be able to maintain structure during concurrent development – more than one developer in a team, possibly editing the same JSP at the same time.

Well, however to test a JSP you still have to publish it back to the  origin  server, as far as I know.

Does CSDT provide protection when 2 developers upload the same JSP? Does CSDT allow for a merge of concurrent modifications? If it does not,  we really  need it.

My understanding is that if one developer check out a jsp, and another developer checkout the same jsp from the same server, when they publish their changes they can still overwrite their work each other.

Am I wrong here?

It would be very useful if I could publish my jsp on a separate server (a local jumpstart for example) for testing. But in current incarnation of CSDT, I cannot, since it is limited to one server.

To address those development issues, using my  home grown Element Deployer  I can assign a jumpstart to each developer. Each developer checkout the jsp from the repository (not from a centralized fatwire), merge his changes the codebase, using standard version control techniques, and then deploy all the  code in his local jumpstart for testing.

It is easy. It simple. It works. And it does what everyone else in the J2EE world takes for granted: you work in your own  local server for testing.

CSDT instead assumes you have only one, centralized server, from where you  get and where you put your code. Not very standard J2ee in my view (although it is a standard Fatwire CS practice, unfortunately)

2. As to your question “(you wonder: why???)”, the answer is simple: CSDT is primarily an export/import tool; the Eclipse part is just a small part of it, and is not even required. Furthermore, CSDT is designed to be portable between IDE’s.

Fine, but here you are saying "CSDT is not really integrated in Eclipse, is is just hosted in it".

This means you cannot use your IDE version control tools, defeating the whole point of using the IDE.  Furthermore the directory layout where CSDT stores his content is "opaque" and you have to do reverse engineering to discover where is it. Also file names are generated by CSDT.

I do not feel  confident I can safely version controls files that I do not manage (it is the tool who decide where things go and their names, not me).

Using my basic "ElementDeployer" I just create files in Eclipse, that can be edited as normal JSP files, I can version control them with the IDE.

I really do not see the point of treating Fatwire JSP as special. They are not, and they can be edited as normal JSP (provided you tell eclipse where jar and tld are, and this is just a matter of creating a link).

In this matter, the old FatClipse was doing a better job, and I just using his technique: edit Fatwire JSP as normal JSP (just adding some jars and tlds to eclipse via a link).

3. Finally, regarding seeing changes immediately upon save – CSDT actually does that in your local environment (you can also use the embedded browser in eclipse refreshed using Ctrl+R, which some people find extremely convenient).

I think you are missing the point here: what I mean is that with my ElementDeployer I just save a jsp, and the result is immediately DEPLOYED on ContentServer (but I am assuming it is deployed in my local jump start, please note, not a shared CS)

Instead, deploying with CSDT is a long sequence of click. When you are working on fixing templates, you need to deploy the code very often .  With CSDT I have to deploy the code, with ElementDeployer I have not. It is immediately deployed on the server.

I am talking of using my local jumpstart server, again. Of course, I agree that using ElementDeployer with a shared server is very dangerous.

Of course, this is within the J2EE development guidelines of separating development from deployment, so your target system will not be updated until you deploy. People use CI frameworks like Hudson for that sort of work, btw.

If you work in a small team it is understandable that you would like more direct methods of working with the test server. However, hopefully we can agree that after bringing in a few more developers things are bound to become messy.

First, I work with a team of 3, and we have to share code and merge our changes all the time. 3 or 30 do not change too much in this respect: you have to merge your changes with others. I do concurrent modifications.

For this reason, I am saying that you have to work on your local jump start, then commit your code to a code repository. You have to rebuild your  QA server with code taken from the repository. Here you can use Jenkins for the task (ex-Hudson).

Anyway, I hope this didn’t sound like I was preaching or anything. I very much appreciate it when people bring up issues with CSDT. This is what helps us make things better in the future.

I am happy you are explaining your point of view. Honestly I am not sure I understood everything of CSDT and  I may  using it incorrectly.

I am writing here my opinions. I will happily publish your answers and correct my misunderstanding.

However...

Why I still prefer FastStart and ElementDeployer

Shameless plug! So far, I still prefer working with ElementDeployer and my FatStart.

Here is why.

You just need a single java class I call the ElementDeployer,  and a bunch of html/javascript files. All of this is the FatStart project. Benefits:

• you get instantaneous gratification (otherwise called rapid application development)
• you can use  your IDE with its version control system
• you have a solution for concurrent modification problems
• you can use a standard Eclipse (or Netbeans of Intellij) since it does not require any  plugin.
• you can develop also with older CS versions while CSDT requires 7.6 and the WEM installed (that is not a light requirement).

Of course there are  a number of assumptions.

The  more important  is that you Fatwire code is stored in a shared code repository (I use git) but each developer has his own local  jump start.

Every developer need to build a complete copy of the environment, including content model and sample content in his jumpstart.

Well,  you do this  using FatStart, that is basically a Fatwire Project deployer.

So: you create your content model with FatStart, edit it with element deployer, distribute to others using a version control system. A QA system is built from sources and scripts, and from there everything is published to preproduction and/or production.

A new FatGoodie: FatProject

Yesterday I had to create quickly a new Fatwire Project. This time, since I have CS 7.6 I decided to try CSDT (you know, the new Eclipse Plugin for eclipse development). But...

What is wrong with CSDT

Well, the experience with CSDT did not live  up to my expectation. Don't get me wrong: I think that CSDT is basically fine. However, it is not what I was looking for.

Here a few  things I don't like.

First:  it stores content in his own directory, that is NOT the Eclipse project directory. So I cannot really version control it, unless I create a separate repository and use the directory where CSDT stores files. While it is doable, it is a bit annoying (you wonder: why???)

Second annoyance: I configure it once using my local jump start. Now I am unable to change settings.  I want to use indeed a different Content Server (accessible through a VPN). With CSDT once you have configured it, it looks like you cannot change that configuration anymore, unless you create a new workspace. Not sure why, but definitely annoying.

Last thing, I have to manually publish changes once in a while. While definitely not a big deal, it is annoying nonetheless, especially if I have the habit to see immediately  the changes once I hit save in eclipse.

FatProject was born

So, after this frustation I decided to come back to my FatStart. Well, actually I just picked the ElementDeployer, since I do not need the complete packaging power of FatStart. I only was looking for the ability  to use eclipse to edit and  deploying elements in the quickest possible way. Basically an eclipse replacement for ContentServer Explorer, more efficient that CSDT for this basic task (actually CSDT performs other tasks but I simply don't need them).

All I had to do is to create an eclipse project, copy in it the ElementDeployer (part of FatStart) and configure a link to the cs/WEB-INF directory. I also added a launcher script to perform deployments manually.

So basically I created a bare-bone eclipse project supporting ContentServer and  able to edit jsp (with completion!) and deploy code changes when I save it.

I thought that this personal stuff can be useful for others, so I collected the eclipse project and I released it in my FatGoodies collections as "fatproject":

How to use FatProject

Checkout the code from here:

https://github.com/sciabarra/FatGoodies/tree/master/fatproject

Then import the project  in Eclipse. If you  have CSDT already configured, that is all; otherwise you will have to set up the eclipse variable "CS_CONTEXTDIR" to point to your local jumpstart "cs" directory.

Now, create your elements and templates. You have to follow a couple of conventions for naming them (see the doc). Then  you can just launch the ElementDeployer class.

It will look for an ElementDeployer.prp file to read its settings then it will  publish jsp (with extension .jspf please) to the Content Server you configured.

Then keep it running. Every time you change a file it is immediataly redeployed.

This is basically all I need to code with Fatwire if I don't have to distribute a complete package (for which FatStart is better suited).

Using this simple FatProject I am in a productive loop: I edit the content model in Fatwire and I save it, then I edit the code in Eclipse and I save it, and I am done: just reloading the page (well, don't forget to disable or clean the cache, anyway)  I can see the results of my changes.

And I am happy!

(It took more time to write this blog post than to build fatproject actually)

Continuos Deployment in FatStart 0.3

Some years ago I was using Fatclipse for the development of a large italian government web site. I liked Fatclipse because using it I was able to develop with Eclipse (with nice completion for JSP tag and Java method ). Fatclipse had also a very nice feature as well: immediate deployment of JSP. Just hitting "save" was enough to get the code immediately deployed on ContentServer.

When time passes, things change. Now I am developing a large UK government web site but Fatclipse is virtually dead (I was unable to run in with the latest 7.5.x Content Server) and since I work on a Mac I do not have the luxury of using ContentServer Explorer (well actually I want to avoid the punishment of having to use it :D, but this is another story). Furthermore  the new CSDT (not yet tried acutally) requires he 7.6 (and I don't have it in my current project).

In the meanwhile I developed (for my own use) FatStart, that is basically a javascript installer for Content Server that can  install elements;  I extended to support XMLPost to upload the whole content model.

I developed FatStart as a packaging tool, to release a website called FatPhone. But  I found to be useful as a development tools, so for my latest projects I dropped CSE (finally! :))  and I am developing using FatStart on a Mac… And I am happy.

Well, almost. FatStart gives me the freedom to develop on my Mac machine,  allows  code sharing with other developer using  a version control system. But since it is a Javascript based installer, I had to "deploy" manually each template or cselement after a change, manually, with the browser.  I just need to click the "install" button, but it still very tedious.  Not very different from what you do with CSE, but tedious nonetheless. Every time I change something, before seeing the result I have to go on the Javascript installer, LOCATE the changed file, and deploy it.

It is a small step but annoying enough because it has to be done after every change, and I was missing the simplicity of Faclipse: just saving was enough to upload the changes code to FatWire… But suddenly I realized I could have also this with FatStart!

Making FatStart easier to use

I had just write some Java code that monitors jsp files for changes, and when it finds a changed file, upload it automatically. So one night, when I was too bored to write code for others, I decided to write code for myself and I extended FatStart adding this feature. Well, I had some code lurking in a forgotten folder that was already doing the upload of something to a CSElement.

Once completed, I commited, pushed on GitHub and it is now on FatStart v0.3 that I released today. You still have to install everything with FatStart as usual at the beginning. But now you can also launch the FatStart continuos deployer executing the script "deploy.bat" (or deploy.sh).

Well, the first time you have to change the config.prp to match your configuration, just in case you are not using a jumpstart in localhost listening in port 8080 or username and password are not standard.

The deployer will deploy all the elements. Note: it is not a complete deployer (although in the future could be), it just deploys ELEMENTS, not assets. So if you haven't created the Template and CSElements, you won't see it in you site. You can see the elements in the element catalog with CSE.

Also note that in current implementation it deploys ALL the jsp (files with extension .jspf) that it can find under folders CSElement and Template (FatStart deploys only the elements that are listed in the install.html). So be careful, you could overwrite something just dropping the wrong file in the directory.

After the first deployment, the deployer  keeps running and monitor work directories. When it finds a changed file  (a file whose  last modification date has changed since its previous visit) it redeploys the file immediately.

Keeping the deployer  up and running while you code has the net effect you can now edit the jsp with Eclipse and when you save it, within one second the file is deployed in ContentServer . An effect very similar to what I was having with FatClipse: just saving was enough to upload the file in Fatwire.

PS of course, if you don't want the extra step of having to clean the cache, remember to put the property "cs.nocache=true" in your futuretense.ini.

PPS. If you also have to code an URL assembler, a filter or other java code, add JRebel to your jump start kit, and you can have a complete no-deploy environment where you can change you Java code, your JSP code and get the result deployed with no intervention. Almost as luxurious as using PHP in the first place! Ok I am kidding, PHP is not that great as a programming language but it has some advantages... well it had I should say now...

Released FatStart 0.2

I just released FatStart v0.2.

New features:

• A new tabbed interfaces - the old single page interface was clumsy to navigate

• The ability to deploy each asset separately on the fly just clicking the install button.

See below for a screenshot of  his new look.

Improving the FirstSiteII URL Assembler

Recently, in the independent Fatwire Developer Yahoo Group someone was asking how to get symbolic URL instead of numeric ones. Since I have a good experience writing URL Assemblers, I decided to write a blog post about improving the more commonly used  URL Assembler: FistSiteII url assembler.

FirstSiteII url assember and its limits

When writing a custom URL Assembler for FatWire, the normal starting point is the FirsSiteII URL assembler that is provided by Fatwire as an example.

A common complaint about the FSIIAssembler is  generated URLs looks like this:

http://localhost:8080/cs/Satellite/FirstSiteII/FSII/Product_C/123445678/12345679

They are certainly more friendly than the "c=Product_C&cid=12345678&p=12345679"  generated by default (using the QueryAssembler, actually), but they are not yet very useful for SEO, since there  meaningful informations  required by search engines are still missing from the URL.

So I will show in this post how to improve the FirstSiteII URL assembler to replace numeric ids with symbolic names, in order to get URLs like this:

In an ideal world I can released the full changed code for the fixed URL Assember, but as far as I know this code is  proprietary, so I cannot release it with my modifications. If someone from Fatwire is willing to give me the authorization, I will make all the code available.

In the meanwhile however, I wrote an helper class where the bulk of the code goes, and I can release it os github since it is all my code. The code is  now part of the FatStart project, and it is available here: AssemblerHelper source code.

So basically, getting that class and following my instructions you should be able to easily duplicate my efforts, as long as you can get the FSIIAssember source code from Fatwire (and as a regular Fatwire customer, you can just ask the Support).

What AssemblerHelper does

The underlying idea of the AssemberHelper is replacing cids with asset names. So for example, instead of using the cid 1234568 for the page named FSIIProducts,  in the url there will be exactly FSIIProducts. This replacement can be applied (or cannot applied, depending on your preferences) to the parameter p as well. I have done it but it is not strictly required.

AssemblerHelper provides 2 static methods:

• name2cid(c, cid)
• cid2name(c, name).

The "c" (the current asset type) is  always required since you need to query for the "name" field by the "cid" in the "c" table. When you do the reverse, it is still a lookup in the "c" table for the id of a rows with the given "name".

Actually, uniqueness of names is no more enforced in all the versions of Fatwire, so there is the danger that a cid cannot be replaced with an UNIQUE name. So in the code there is a provision that when a name is not unique, no replacement is done (name2cid will return the numeric cid instead of the corresponding name).

The resulting name is also URL encoded, so if can be used in URLs. Furthermore, since there is the danger of SQLInjection, I added some code to detect quotes and to filter out dangerous URL to reduce the risk.

The decoding method (name2cid) will translate a symbolic name back in as a numeric c, searching for the id of an asset of the given type with that name. The method will take care of the url decoding as well.

Adding the encoding to the FSIIAssembler

Once you have the AssemblerHelper code, you got the FSIIAssembler code, there are only a couple of modification required in order to use my AssemblerHelper.

In the getPath method the url is built concatenating some elements. The current cid is stored in data.cid, the current p is stored in data.p, and current c is stored in data.c

Locate the code that builds the returned path (it is easy, as long as you can read Java) and  replace references to data.cid with AssemblerHelper.cid2name(data.c, data.cid) (you need an"import com.sciabarra.fatwire.*" in your source code as well).

If you want to replace also the ∫with a simbolic name, you have to add also AssemblerHelper.cid2name("Page", data.p).

That is all for the encoding.

Adding the decoding to the Wrapper Page

Fatwire actually do the decoding in the SatelliteServer. Since the decoding requires access to the database, it cannot be performed on remote Satellites. For this reason,  the actual  decoding of the name back into a cid is performed in the wrapper page.

However, before adding the decoding in the wrapper page, an additional modifications must be done in the FSIIAssembler. There is a check that cid and the p are actually numeric in valueOf method of the inner class FSIIAssemblyData. Since now the cid and p can be string, that check must be removed. So, just locate the exception surronding the Long.valueOf(cid) and comment it out. Do this also for the Long.valueOf(p) if you have replaced also the p with a symbolic name.  That is all for the URL assembler.

Now it is time to add the conversion from the name to an id in the WrapperPage. Open the FSIIWrapper CSElement and look for the render:satellitepage tag. Replace arguments

<render:argument name="cid" value="<%=ics.GetVar("cid")%>"/>
<render:argument name="p" value="<%=ics.GetVar("p")%>"/>

with

<render:argument name="cid" value="<%=
com.sciabarra.fatwire.AssemblerHelper.name2cid(ics.GetVar("c"),ics.GetVar("cid"))%>"/>
<render:argument name="p" value="<%=
com.sciabarra.fatwire.AssemblerHelper.name2cid("Page",ics.GetVar("p"))%>"/>

And now you are done.

Notes on performance

This solution is a potential performance killer since it performs a query for each URL requests. Fatwire has a ResultSet cache, so this problem could be not an issue, but if it is (maybe because there are too many different requests so the query cache cannot cache enough result sets) my suggestion is to add an additional caching level, storing name2cid mapping in an HashMap. I would use a WeakHashMap (so memory won't be an issue) and store the hash map in the application context, but I have not tried this solution so far. It is left as an exercise for the reader...

Released FatStart: Eclipse based FatWire development kit

Sometimes you are a victim of your own success. In the last few weeks I had so much to do that I had not the time to post anything. But since I started a couple of new FatWire projects, this gave me the opportunity to release another open project I was thinking of: FatStart.

What is FatStart?

Basically, FatStart is a template for FatWire Content Server site development that solves a few very well known problems in FatWire development:

• It allows you to use eclipse (without plugins!) as a development enviroment  (instead of the awkward ContentServerExplorer!)
• It let you store the source code AND the structural assets under a version control system like subversion or git.
• It allows many developer to collaborate having each one his private FatWire Jump Start instance
• It allows to deployment  a website on a production FatWire server without resorting to publishing

The trick is that you store structural assets in json format, write your jsp in eclipse then publish them easily at any time using a javascript installer, both in your local development Jump Start or in the production server.

I know that there are already 2 Eclipse plugins for Fatwire development, but they are far from satisfactory. My solution tried to grab ideas from them but I am using an unmodified eclipse (no plugins) and just javascript to deploy the content in FatWire. The result works pretty well.

The project is basically the same distribution kit I developed for FatPhone but it is now standalone and I provided also the eclipse configuration files.

How FatStart works?

Just to give a taste of how the system works, please give a look to a few screenshots below.

Once you checked out FatStart from the GitHub repository and imported it in eclipse (also you have to do some tweaking explained in the README), this is what you will see:

Now you can edit your jsp directly in eclipse. You can also create your content model directly in Eclipse in Json format. For example this is the format of a content definition:

{
 "AssetType": "fsContent_CD",
 "internalname": "fsArticle",
 "internaldescription": "FatStart Article",
 "parentselectstyle": "selectboxes",
 "RequiredSingleParentList": "fsCategory",
 "RequiredAttrList": "fsText",
 "OptionalAttrList": "fsAbstract"
 }

The required fields are  described in the developer manual, in the section related to XMLPost of FlexAssets.

The work done can be deployed at any time in FatWire for test  calling the installer: 

http://localhost:8080/res/install.html. 

For example if you change the JSP of  the fsLayout template, you can deploy this way:

If you instead change a content definition, you can deploy it as well this way:

That is all.

 The installer is incremental and will replace existing assets and elements with the new version.  

FatStart Benefits

Basically, since all the site configuration is stored in source files, it can be stored in a version control system.

Each developer can have his own version of the site, and when something else change the code, he has only to update the source code of current version, using common merging techniques, and deploy it in his JumpStartKit.

Once the application is ready, the deployer have only to checkout the source code and install it in the production server.

There are a few limitations. So far the installer can only deploy assets and templates, but it does not create sites, flex families, treetabs or users and roles. Those configurations should be performed manually, although it not a big deal to do so. 

So far the system is in use in a couple of sites I am developing and is robust enough to be used for day to day development. Anyway it is still an alpha release so I recommend precautions (although, once the assets are created in Fatwire, there is no so much differences from creating them manually). 

Any comments is welcome and I recommend the group FatwireDev I manage to discuss it.