WebCenter Sites 12c: step by step installation

Transcript:

Welcome. In this video we are going to see how to install Oracle WebCenter Sites 12c.

Before starting let’s check the prerequisites for operating system, database and java runtime. We need a supported operating system, and we will use Oracle Linux version 7. We need a supported database, and we choose, with no surprise, Oracle. Finally. we need a Java Development Kit, and we are going to use the latest available when I writing this.

This procedure can be executed in a physical server but for the purpose of this video, I am using a Virtual Machine, and I used VirtualBox. So I went on the Oracle website, and downloaded VirtualBox. I used VirtualBox for Windows, but there are versions available for Linux and Mac too, and they work in the same way.

A virtual machine alone, however, is an empty box. You can install everything from scratch, but it a time consuming task. Luckily, you can download an image, which already included operating system, and database. In the Oracle website is available an image, which fits our purposes for installation Sites: it is the Oracle Developer VM. I am not showing here how to download and install VirtualBox and the Image. However it is pretty simple and obvious process.

The only problem has this image, is the outdated JDK. It does not fit the requirement of a supported Java Development Kit. So the first step is downloading and installing the latest JDK available for Linux. You can directly download and install it from the browser. The system will ask you for the root password, in order to perform the installation.

Now we can proceed installing Sites. We go to the Sites download page, accept the agreement and we download it. See the video for the location. You need to authenticate to be able to download.

After Sites, we need to download the Weblogic application server. Note the location in the video. For Sites, the enterprise manager is a required component. The recommended download is the fusion middleware infrastructure.

Last but not least, in order to install the analytics features we download also Endeca server. We will not show Endeca installation in this video, however.

Let’s wait for the download. All the files are distributed in zip format, so we have to extract all of them. For simplicity we unpack everything on the desktop.

Now we can start installing. First step is to install the middleware infrastructure. We install from the command line. We first check we have java in the path and the version is correct. Then we launch the executable jar with the command in the video. The installer unpacks itself, then finally shows its user interface.

Steps to perform are simple: skip the patch install because we do not have any patch.

Then select the installation folder. We are going to use a Middleware folder in the home directory.

Finally select the first option, since we do not want the examples of the infrastructure.

The installer will ask for your email address then it will complete the installation with no intervention.

Installing Sites is similar. We go in the installation folder, and we execute the executable jar containing webcenter sites, as you can see in the video.

After the installer extract and launch itself, you are asked again for patches. You can skip this again.

Then it asks for the folder, where you just installed the Middleware.

Finally, there is an option asking what you want to install. We will install Sites with examples. Now, just watch the installer to complete the installation.

Once the software is installed, we need to create some schemas in the database. We will use the Oracle Repository Creation Utility for the task.

Open the terminal and change to your main installation directory. The R C U is located under oracle_common/bin

You need to start R C U, then communicate where is the database server we want to use.

Now, let’s select we want to create a repository.

You have a form where you have to provide the details of the database server. In the virtual machine it is located in the same host. So we give localhost, with the usual port 1521 and the service name O R C L.

The virtual machine we recommended has a default user sys with password oracle. The R C U connects to the database,

Once connected it offers a selection of the schemas we want to create. Of course we select the webcenter sites schemas.

Next step is to provide a password for the schemas. This password will have to be provided in the next step for connecting Sites to the created databases.

Now just sit down and watch the system to create our schemas.

To install Sites you actually need to create a weblogic domain, using a template provided with the installer. We will use the configuration tool for weblogic for this task.

Open the terminal and change to your main installation directory. The configurator is located under oracle_common/common/bin. Note this is a different directory than the R C U utility.

Start the domain configurator, then select you want to to create a new domain.

I am giving sites as the name of the installation directory.

I am going to select as an option the WebCenter Sites template with examples, then as components Visitor Services and Insights. Note in this video we only show how to install Sites.

I am giving a password for the weblogic enterprise manager. Note this password must be used to start the services.

Now we have to configure the database to use. Here the system will retrieve the schemas we just created in the preceding step. We need to provide a password, the password we used when creating the schemas.

We are offered to use the schemas we created with R C U . We just need to accept them.

Now the system will verify the connection to the database. This step is very important.

We can now configure the machines and assign them to servers. Without assigning machines we will be unable to start the services for completing installation. So we choose here to assign all the services the local machine. You may do differently if you have more machines.

That is all for the domain creation.

We are ready now to start servers. We have actually to start 2 services: the node manager and the administrative server. Those 2 are required to actually start and configure sites.

To start services, go in the bin folder of your domain directory. Now, first start the node manager as a backgroud process. Afterwards, in the same directory launch the administrative server.

Now an important step: you need to give to sites a special authorization. To do so, first change to the folder W C sites/bin under the domain folder. Then execute the script grant O P S S permission.

You need to provide the weblogic credentials you set when you created the domain.

I also create a shared folder. Note that in production systems a shared folder may be placed in a N F S.

Now, access to the enterprise manager, using the URL shown in the video, then start Sites.

Check now that Sites is up and running, invoking the Hello C S path.

We are finally to the sites configuration, the last step. Access to the sites-config-setup path.

Click on Begin and select the shared folder you just created.

You can use the default for the J N D I name.

Now specify the I P and the port of the main entry point of WebCenter Sites. Since we are using a virtual machine we use its I P.

Here we specify the entry point of the C A S, and we will use the same I P as before.

For a development environment we can leave the default passwords. Of course they must be changed for a production installation.

We have almost finished. Click on start to configure.

You can just watch the configuration to complete. Sites will be ready after this step is finished.

Let’s verify if we can login in Sites to check if everything is installed.

Thank you for watching!

AgileSites Meme Collection

Just for fun, here is a collection of AgileSites memes…
Vote your favorite!

AgileSites, the real story

A frequent question people ask on AgileSites is: why you did it?

Well, it was not really a matter of choice. It was a matter of necessity. If you want to know, I will tell the story.

The tale of a Fatwire project

Once upon a time, it was something between 2007 and 2009, we were contracted to implement a web site using Fatwire (the precursor of WebCenter Sites), then hand it over to the customer for maintenance.

The customer provided the HTML mockup,we had just to code the jsp based on that mockup.

So we did our work, using just plain JSP as it was the standard at the time, and we built the content model, mostly working in a JSK that then we published in customer Development server.

Once finished, it was supposed to be all.

I was wrong. After the delivery to the customer, they first started to change the code, then they kept calling us back for a long list of different tasks to help them to add a lot of other features they decided to implement.

The problem is that our suppor work doing those changes became an unbelievable pain because of the limitations of the enviroment.

The nightmare of updating the html

The first annoynace came from the HTML. The original mockup was changed. Actually the customer changed the mockup multiple times then he gave it back to us for implementing it back in the JSP.

So the first problem was: were the change happened? Once you have implemented the JSP, the mockup has been split in many difference pieces and lot of code has been added.

So we had to use a version control system, diffing the html, the patiently trying to locate the JSP where the change happened. Very often the html was changed structurally so the changes were spread in many different templates and we ended up, multiple times to have to rewrite many templates. The logic and the html were mixed up and that is totally wrong.

After this experience I understood that HTML must stay HTML in the original form, and Java must stay Java with no markup. Mixing html and code is the biggest mistake in the history of internet development and it is acceptable only in those rare cases (today almost disappeared) when the HTML and the Java coders are the same person.

People don’t read manuals

Part of the work was to support the development done by other people. I cannot believe to what I saw. Even if our code was done properly, with all the correct tags used according best practices, maintainer did not even care to understand.

They did a quick look to the tags, and they found one tag they undesrtood: ics:sql. After this, all they did it to throw SQL queries everywhere, using then a lot of embeded java code to render queries.

When I asked why they did it, they simply said they did not understand the complicated API so they had no choice. Actually I have to admit that ics:setasset or assetset:getaatributevalues are definitely complex, but I never realized they are a real stopper for some people.

There are people simply they cannot use a system where there are many conventions and calls with multiple parameters. So I realized what they really needed an api very simple and immediate.

Something like:

Asset a = e.getAsset();
a.getString(“Title”)

So I started to wonder how to make a library simple as this.

Java java java everywhere

JSP are supposed to be a sort of “extended html”, and you are supposed to perform your tasks using mostly only the tags. In practice, you always need to write complicated logic, and using tags for this is simply not an option. So you end up opening a scriptlet… and your entire JSP became a large snippet of embedded Java code.

Except, you soon realize that the embedded Java within a JSP is NOT full Java. You cannot create a class and share it. So you cannot even have a method used in 2 different templates, you need to duplicate the code. The biggest advantage of Java (code reuse) is simply not available.

Indeed this is what many Sites implementations are: a giant copy and paste of the same code repeated with modifications spread everywhere. A real maintenance nightmare.

The site kept breaking

Take a team of untrained developers, working without a good knowledge of the system, using a shared enviroment, working with copy and paste programming, and having to apply a mockup that is changing all the time. What you get? Answer: a giant number of issues. What is worse at every change you can break something so you may end up having to retest the entire site.

The customer kept asking me again and again: is it possible to automate tests? I wished all the time I could have answered something better than “yeah, you can use selenium”….

Moving back and forward the content model: oh my

The customer kept changing the content model. Basically with no planning: every time they needed someting they added an attribute somewhere, with a lot of duplication of content too. Then they hacked the code to use it.

Since we were forbidden to use their development server, we kept wasting time exporting their model in a copy (a Jump Start) were we could do the development. The process was more time expensive than doing everything else. I think we spent 2/3 of the time moving content model back and forth. The problem was also that the customer refused to acknowledge that time as “coding time” so the process made us to actually lose money.

Here I realized in no way you have to do this. A reference enviroment where people changes things and then copy back is basically conceptually wrong. You need a development enviroment in local machines for every developer, then use some build script for deployment.

But never apply change in a shared enviroment. Sooner or later that enviroment will became unaccessible leaving you in the dust.

You really need a build

It is actually a bit more that just moving the content model. The entire system is comprised of many different pieces: extension code, imported elements, xml configuration files and so on. Putting all of this manually is really like having to compose a puzzle every day.

We also had a lot of deployment problems because the many different pieces. Assembling the whole reading a document is awkward and really dangerous. You need a build script and a setup script. You need some code that will take care of automatically deploying all the pieces. Without it, you will end up forgetting pieces and then spending hours to figure out what is missing.

AgileSites =
Putting all the pieces together

After this nightmare, I was determined to create something we can use for ourselves to save our life from all the problems that plain Fatwire (and Sites) poses. So I created something that:

1. really separates HTML from Java
2. allows to use full Java, without any limitation
3. it has a simple api for starters and common tasks
4. it has an installer for automated build
5. it has support for automated tests

AgileSites was born! How it was applied to real world project is another story, I will tell in another post.

A powerful command line interface to WEM api

Are you working with Fatwire/Sites REST API, commonly named WEM api? We are currently working with it a lot, building AgileSitesNG. 

The API is pretty complicated so we need a lot of processing to locate the informations we use. Furthermore, a common annoyance is getting an authentication token for each request. You can generate a token manually, with a couple of http call, but in the end working with it became awkward and time consuming.

So I started to dream a command line interface like “wem” where I could just select the call I wanto to  get the whole json output well formatted. For example, I wish I could do this:

./wem sites

{ "schemaLocation": "http://www.fatwire.com/schema/rest/1.0 http://localhost:8181/cs/schema/rest-api.xsd",
  "total": 2,
  "startindex": 0,
  "count": 2,
  "site": [
    {
      "href": "http://localhost:8080/cs/REST/sites/AgileSitesNG",
      "id": 1000000000000,
      "name": “AgileSitesNG",
      "description": “AgileSites NG"
    },
    {
      "href": "http://localhost:8181/cs/REST/sites/AdminSite",
      "id": 1253041378564,
      "name": "AdminSite",
      "description": "AdminSite"
    }
  ]
}

Even better, when I have such a complicated output, Iwould be able to filter part of the content. For example in an ideal world so I could get the list of the users with its roles just typing:

./wem sites/AdminSite  ".siteUsers.user[]|{name,roles}"

{
  "name": "fwadmin",
  "roles": [
    "AdvancedUser",
    "WorkflowAdmin",
    "SiteAdmin",
    "SitesUser",
    "GeneralAdmin"
  ]
}

The wem script

Actually the samples below are not a dream, they are the actual  what you have seen before is the real output of the following script (for Linux and OS X).

#!/bin/bash
CALL=${1:-sites}
HOST=localhost:8080
USER=fwadmin
PASS=xceladmin
REQ=$(curl -XPOST -sD -  -d "username=$USER&password=$PASS" http://$HOST/cas/v1/tickets -o /dev/null | awk '/Location:/ { print $2 }')
TOK=$(curl -s -XPOST -d service=\* $REQ)
curl -s -H "Accept: application/json" "http://$HOST/cs/REST/$CALL?multiticket=$TOK" | jq "${2:-.}"

You can put the script in your local directory or copy the script in a /usr/local/bin/wem file, then execute a chmod +x. Then, most important ensure you have in the paths the commands curl and jq.

While curl is common and available everywhere, jq is less common but you can download it from here http://stedolan.github.io/jq/download/.  If you are in OSX and you have homebrew installed you can just type “brew install jq”.

Actually the scripts works fine also on Windows provided you have installed a bash shell, that is provided in counless packages (cygwin, cmder, msysgit etc etc), curl and jq

Note the script must be edited to point to your current installation of Sites. If you have a Jump Start running on localhost with default users and passwords, then you are fine. If it is not the case, the script  must be adapted (changing the HOST, USER, PASS variables of course).

Wem Script syntax

The script takes 2 parameters. The first parameter is the Sites API you want to call, as listed here: (without the “/“ prefix). Some examples:

./wem sites
./wem types
./wem types/CSElement
./wem sites/AdminSite

The output is pretty verbose, so it helps using the “jq” tool syntax that is pretty handy.

The second parameter is indeed the a “jq” selector. While you can read the complete manual, a few examples are:

Selecting only site information:
./wem sites ".site[]"

{
  "href": "http://localhost:8080/cs/REST/sites/AgileSites",
  "id": 1000000000000,
  "name": "AgileSites",
  "description": "AgileSites NG"
}
{
  "href": "http://localhost:8080/cs/REST/sites/AdminSite",
  "id": 1253041378564,
  "name": "AdminSite",
  "description": "AdminSite"
}

Selecting only attribute values:  
./wem sites “.sites[]|.id,.href"

"http://localhost:8080/cs/REST/sites/AgileSites"
1000000000000
"http://localhost:8080/cs/REST/sites/AdminSite”
1253041378564

Rewrap the selected information in a json map:
./wem '.sites[]|{"id":.id,”href:":.href}'

{
  "id": 1000000000000,
  "href": "http://localhost:8080/cs/REST/sites/AgileSites"
}
{
  "id": 1253041378564,
  "href": "http://localhost:8080/cs/REST/sites/AdminSite"
}

Those simple examples should give an idea of the power of the tool. Enjoy.

AgileSitesNG sneak preview

So, what are we doing with AgileSitesNG?

A few things, actually:

• A new “proto”  site, pretty full featured for responsive and mobile development.
• An development tool chain with on-the-fly compilation of javascript and java sources, including hot deploy of jar and live reload of javascripts
• A revamped architecture for AgileSites leveraging AngularJS features 

What is in the Proto site

Now, instead of a simple demo we are introducing a complete mobile responsive site based on BootStrap and AngularJS.

Here is a snapshot of what you can see with the  Proto site: 

The site is actually entirely content managed with Sites:

The development chain

Developers can work using their favorite IDE (for example IntelliJ IDEA in the snapshot) and they can code using directly Javascript, HTML and CSS. 

Actually the system integrates preprocessing of less, coffescript and other javascript related technologies, concatenation, minification and compression:

The build system is totally integrated and interactive: editing a “less" or “coffescript" file will trigger compilation to javascript, then the optimization  of the web assets, producing the best output even while the development. A nice addition is Live Reload for Javascript. Also the usual AgileSites Hot Deploy still works. 

Basically developers can now see instaneously the result of changes to javascript, less files, html and java code without even having to hit the reload button in the browser.

But actually there is more, since even the content model is now automatically generated.

The AgileSitesNG Architecture for Angular

Traditional architecture for javascript  works using a page-based approach, with an MVC server based. Angular however is a client-based MVC so the traditional approach does not work anymore. Instead the server need to provide 3 things:

• Page Snippets, invoked by Angular when needed
• JSon services that fits the Page Snippets
• A Navigation that can be configured by the CMS

So, the way AgileSitesNG works is the following:

Developer write their code using html code snippets, like those:

The snippets are automatically imported but what is most important is that after the import:

• Content Model required to render the website is automatically generated from the template

• JSON services to render those templates are also automatically generated from the Content Model

• Navigation for the client side application (the routing rules) are generated by Sites site plan.

In the end, developers only need to think to code their client side. The backed support for content managing a site built  Angular is generated by the AgileSitesNG framework.

WebCenter Sites in the cloud: unattended installation

You may want to deploy  your WebCenter Sites in a cloud (even a private one). This is a common requirement nowadays. We have done it a number of times, and  I am sharing here my experience building this thing.

What you need is an automated installation, and you can do it using the Silent Installer part of Sites. In this post I describe the procedure using Apache Tomcat.

I assume you have an Oracle database already deployed so I won’t cover the details of installing Oracle.

Downloading the needed software

Step 1. Download and unpack Apache Tomcat. First and before all you need to go in the Apache web site and download Tomcat

Once you downloaded it, unzip it in /opt/oracle/sites

Step 2. Dowload and unpack WebCenter Sites

Now you need to go in the Oracle Web Site (or in the Oracle edeliver site) and download the WebCenter Sites. You have to unzip twice: fist unzip the downloaded file, then locate the WCS_Sites.zip and unzip it again under the /opt/oracle/sites folder.

Step 3. Install the ojdbc6.jar and the tools.jar.

Now you need the Oracle JDBC driver. Either get it from the Oracle distribution or download from the Oracle WebCenter Site.

Once you downloaded it you need to place it under the apache-tomcat-xxx/lib folder in order to use it with Tomcat.

Important: you also need to put in the class path the javac compiler. So copy $JAVA_HOME/lib/tools.jar in  apache-tomcat-xxx/lib

Let’s review if everything is fine. If you followed my directions you should have now this situation:

Check also that in the apache-tomcat-xxx/lib there are the files ojdbc.jar and the tools.jar.

Step 5. Configure Apache Tomcat.

Edit the /opt/apache-tomcat-xxx/bin/catalina.sh file and add at the beginning, after the #!/bin/bash line:

export JAVA_HOME=/usr/java/latest
export CATALINA_OPTS="-Xms512m -Xmx2048m -XX:MaxPermSize=192m -Dfile.encoding=UTF-8 -Dnet.sf.ehcache.enableShutdownHook=true -Djava.net.preferIPv4Stack=true -Duser.timezone=UTC"

Note:  you may change the path of the jdk and the memory constraints to values more suitable to your system

Configuring the Silent Installer 

Now, we are ready to configure the silent installer.

Step 6. configure the silent installer located   Sites/install.ini and change it:

loadfile=/opt/oracle/sites/misc/silentinstaller/generic_omii.ini

then edit the file /opt/oracle/sites/misc/silentinstaller/generic_omii.ini  and make the following changes:

sCgiPath=/cs/
CSInstallDirectory=/opt/oracle/sites/home
CSInstallSharedDirectory=/mnt/shared
CSInstallAppServerPath=/opt/oracle/sites/apache-tomcat-7.0.57
CSInstallDatabaseType=Oracle10
CSInstallDBDSN=csDataSource
CASHostName=localhost
CASPortNumber=8181
CASHostNameLocal=localhost
CASPortNumberLocal=8181
CASHostNameActual=localhost
CSInstallWebServerAddress=localhost
CSInstallWebServerPort=8181

Note 1: here I am using host localhost and port 8181 for test - you have to change to the actual hostname and port where the server will be available
Note 2: The default configuration will install all the examples. If you do not want then, change all the properties with “Sample” in the name to false.

Step 7: Create the database

Access to the database server, and create the user in the database. For oracle, access to the SQLPLUS command prompts as a dba  and execute those statements:

SQL> create user CMS identified by CMS;
SQL> grant connect, resource to CMS;

Note: this is a very simplified example. In real world you will of course use a better password, and also you will allocate a table space for the user. But here I am only providing a sample.

Configuring Tomcat

Step 8: Configure the database access in tomcat

For tomcat, you have to edit the apache-tomcat-xxx/conf/server.xml and locate the section:

 
Then add the following code to configure a data source.

Since I am also installing in port 8181, you need to change the port tomcat listens. This is also in the server.xml: locate and change the Connect section accordingly this sample:

Note: this sample  is referring to a local oracle-xe database. In real world you will put your own database parameters. Also, since the oracle-xe binds to port 8080, I had to change it. You may not need this  change if you are using a database in another host.

Step 9: Extend the classpath in tomcat

Since CAS expects to find some files in the class path you need to extend the class path. So go in apache-tomcat-xxx/conf/catalina.properties and edit common-loader adding /opt/oracle/sites/home/bin to the class path, as in the following screenshot:

Test it Attended 

Now it is time to test if things will work executing an attended installation.

Before begin, copy this piece of code (a tester for data source connection) in /opt/oracle/sites/apache-tomcat-xxx/webpass/cs/testdb.jsp:

<%@ page  import="javax.naming.*,javax.sql.*,java.sql.*"
%><%
Context initContext = new InitialContext();
Context envContext = (Context) initContext.lookup("java:comp/env");
DataSource ds = (DataSource) envContext.lookup("csDataSource");
Connection conn = ds.getConnection();
%>OK

Now, go in /opt/oracle/sites/Sites and type

sh csInstall.sh -silent

Then the automated installation will start. After a short while it will stop asking to start the application server:

When you see this message, you are not yet done.
Now you need to open another terminal, go in /opt/oracle/sites/apache-tomcat-xxx/bin and type:

./catalina.sh run

This command will start tomcat in foreground (a good choice to test for errors in logs).

Now, first test the deployment has been performed properly calling with the browser:

http://localhost:8181/cs/HelloCS

You shold see the following message (o similar depending on the version you are installing):

Also, test database connectivity using the script I provided before.
Invoking 

http://localhost:8181/cs/testdb.jsp

You should see simpy an OK message (no exceptions or other errors).

If things are ok, come back to the first terminal and press enter. Now the installer will create the database. You will see a number of commands executed. It can take a while

If everything is fine you should be greeted with a message:

Installation Finished Successfully 

Congratulations!

Final Touch: automated installation

Ok so far so good, but the automated installer is not good enough for a completely automated installation in the cloud, because it is attended. So we have to automate more in order to have everthing unattended.

The Sites installer stops when it completes the deployment of the war, asking the user to manually start the application server, wait for the application server to be up and running and then complete the installation when the user press a key.

So we need to write a script to perform the synchronization of the installer with the application server startup. I wrote such a script.

Here is the script auto.sh, split in pieces and commented.
It is expected to be placed in /opt/oracle/sites.
It requires the commands nc  and  wget to run. Those commands are not standard unix but very common and easily available.

The whole script is available as a gist here. This is the script inlined and commented.

#!/bin/bash
cd $(dirname $0)
BASE=$(pwd)
killall -9 java nc wget

It it a bash script, and it initializes changing to the directory where itself is placed; it kills  any process that can be leftover of a precedent run. 

echo "<<< deploying sites"
cd $BASE/Sites
rm out.log
touch out.log
nc -l 12345 | sh csInstall.sh -silent | tee out.log &
while ! tail -n 1 out.log | grep "press ENTER."
do sleep 1 ; echo ...deploying...
done
echo ">>> deploying sites"

After creating an empty log file,  the Sites installer is launched in backgroud; the input is bound to a socket listening in port 12345 using the command nc while the output is sent to a log file.
 
The output in the log file is monitored  in order to know when the installer complete its first run and stops, waiting for the application server to be started.

echo "<<< starting tomcat"
cd $BASE/apache-tomcat-*/bin
sh ./startup.sh
while ! wget -q -O- http://localhost:8181/cs/HelloCS | grep reason=Success
do echo "...starting..." ; sleep 1
done
echo ">>> started tomcat"

When we reach here, the installer has already deployed the war and  it stopped with the “press ENTER” message.  

Now, we can start the application server (tomcat) and then we wait until the application server is up and running.  

We check that the application server is ready, looking for the  “HelloCS” servlet and then checking for the  “Success” string it contains.

echo "<<< installing sites"
cd $BASE/Sites
echo | nc localhost 12345
while ! tail -n 1 out.log | grep "Installation Finished Successfully"
do sleep 1 ; echo ...installing...
done
echo ">>> installed sites"

Now we can complete the work. Using the nc as a client we contact the nc listening as a server, and we can notify the installer it can complete its work “pressing enter” in a socket. 

So the installation can be completed. We keep looking at the output of the installer until the message “Installation Finished Successfully” appears.

And that’s it.

If you need more...

This script works of only with Tomcat and it only deploys one instance.  A real world installation will more likely require the deployment of multiple nodes,  satellites and web server and the installation of custom components.

Sciabarra.com is  experienced in automating larger and complex deployments, using puppet and ansible to pilot them, and build virtual machines for development with Vagrant. If you need help in this area do not hesitate to contact us.

AgileSites for Content Managers: inverview with Rogier Voet

In this post I am interviewing Rogier Voet, a CMS facilitator that was the product owner in an Agile Projet with a team using AgileSites.

Hello Rogier, can you plese introduce yourself?

Hello I'm Rogier I'm 36 years old and worked primarily for websites whole my career with different kinds of roles, editor, community manager, project manager, content manager and the last years focusing on improvement and innovation of the KPN.com website

What is your specialty area?

I'm a bridge between the business, who wants to achieve certain goals with the website (by adding functionality) and the development team. My specialty is making sure that the solution is user-friendly for both people who have to use/maintain it, but also for the end-user who uses the functionality on the website. 

What is your experience with CMS tools?

 In my career I have seen a range of CMS tools, from custom developed (PHP) CMS-systems, to different kinds of CMS-software. Simple ones like WordPress, but also Tridion, (several versions of GX Webmanager), Drupal, Fatwire and the lasted version Webcenter Sites

When you happened to work with AgileSites and what was your role?

I worked with KPN Sites, during Project LEGO, which was a Pilot to demonstrate that you can develop, sustain and innovate a website very fast using Agile, Fatwire and this framework. My role was Project Owner/CMS Expert,  so I decided which functionality's were to be build and worked with the team so the output was both flexible and user friendly. 

Which problems the framework helped to solve?

It was really an accelerator, normally you would much need more frontend development capacity (we only had one frontend developer available), and because of combined code it makes building upon your developed code much harder. Because you separate the work, it's easier to review and test, which is critical part of Agile Working. 

Did you noticed  any differences using it compared not using?

The rate of development is much higher, also it makes it easier for both java and frontend to work together on a project. 

Do you think Agile is a buzzword or can be really implemented?

I think it's both, but I really think Agile can be implemented. It's nothing new, but finally big company's are trying to get it. I worked with small teams for years using this type of methodology.  The key part is independence and communication (feedback).  

What you would suggest to implement moving forward?

I think this framework is already a smart choice if someone wants to improve speed of development, and increase automated testing.  I think it can be improved if the systems also helps more with the importing, exporting and merging code into Fatwire.          

What you love to do when you do not have to work with CMS and Fatwire?

I love to spent time with my girlfriend, friends and family; I love playing board, card and computer games. I like to play golf, badminton and cycling. And I always enjoy reading good books and watching movies and serie

Why automating your Sites tests will REALLY save two thirds of your QA time

Let's face it. The vast majority of Sites project are late already the day when development starts. 

As a result, delivery pressure is high, budget is usually under heavy scrutiny and all the “unnecessary” development tasks are cut out, since d no one is allowed to “waste time”.

Under this so common scenario, let see what usually happens

The low-quality, late project: one years instead of 5 months

In the standard scenario (I would say, 80% of the real world projects I worked on) pressured developers  rush write as much code as they can quickly, while project managers usually plan skipping altogether the task of writing automated test. 

The common motivations are usually: 

•  “because there is no time” 
•  “it is redundant since there is the QA team for that”
•  “We need to double the amount of code to be writter” 

Often there is another motivation, but no one admit it: we do not know how to do it!

Developers are expected to code their part and test it… manually. No test code is asked to be delivered. 

The big problem you face skipping test automation is the QA team needs to do a lot work. Lot more than necessary actually. The most important waste of time is the need to test and retest everything at every release cycle. 

Why? Because in a manual-test-only environment, once a bug is fixed,  it not fixed forever! Experience shows bugs thought to be fixed can pop up again at any moment. Also, new bugs will be introduced in old code, so testers do not trust that old bugs are gone! Since there is no way to know the quality of the whole project, testers end up repeating the whole testing cycle many many time. 

If you look to real world numbers you can easily see that in a non-automated test environment, the QA cycle can takes up to 3 times the development time. So in a project initially scheduled to be 3 months of work, with 2 months of testing, you usually  end up testing and fixing bugs for six to nine months, delivering the project after one full year! 

The unfortunate occurrence is the consequence of cut out any “unnecessary”. Or simply not following a true agile practice (although the rush-to-code development method is often sold as an agile one!).

The high-quality, on-time project: 6 months of build-on-quality project

Test automation however, one of the key features offered by AgileSites, is something that can completely change the landscape of your project, keeping it on track even under heavy pressure constraints. 

Assume that, even if you have strong time constraints, you can manage to convince the customer to add to the 3 months of planned development time, and to 2 months of testing planned, one extra month allowing developers to actually write automated tests for their code. 

I am well aware that that this is a very difficult task for many project managers! I also understand that normally writing tests in sites is not easy. Actually, allowing code testability is the main reason of the existence of AgileSites ! Someone suggested that AgileSites should  be instead named TestDrivenSites.

It is also true that writing tests can be seen as an additional burden to put on the shoulders of the developers. So instead of taking 3 months of developments, with appropriate tools it will take maybe 4 months. This can be seen as a waste. It is not:  developers are the best people to write automated tests for their code.

Why developers should write tests

You may ask: why should developer write tests? There are the testers for that! That is, unfortunately,  an inaccurate view of the development process. First, developers usually DO test their code. They need to. No developer can write a piece of code and produce instantaneous correct code. 

Every developer, after writing the code, they verify if it works. At least if it works at least a little bit… The problem with code manually tested is that the test follows an an ad-hoc, unrepeatable and manual process, and is performed in a limited development environment.

As a result, developers ends up testing  their code in an unrealistic environment, without integrating their work with the work of others, often limiting to test just a few use cases, and then they throw away forever this significant amount of work! Not writing automated tests usually means to throw away at least half of the development time.

Also a developer is the only person in the world to know really need to be tested, since they wrote they code. They know what they should check. They can look at their code like in a fishbowl and check the fish is doing what is expected. On the opposite, testers are looking in a black box, they have no clue what the code is doing so much of their tests are an educate guess and they may test what is actually constant, so it cannot change! So they waste a lot of time testing unnecessary parts.

Surprise! The extra investment pays off. Hugely.

If you really allows developers do that, benefits are huge.

First, code delivered to the QA teams is in a much better shape.  Basically it is already largely tested and mostly works.  So the QA team must check for real bugs, not for integration issues, overwritten code, forgotten features. 

Second, because there is a test suite,  you do not have to run an entire test cycle for every releases. In such an environment, the QA cycle is very fast. The QA team can test the work, then produce bug reports that are sent to developers. Developers will fix the bugs, but because there is a test suite, they can be sure they are not going to introduce new bugs. 

Third,  the QA team can be confident of the safety of the fixes, and usually they will run the full test life cycle only twice: the first time, and the last, before the release. Using automated tests, there is confidence that only new things must be tested, and not everything.

So just adding one month of extra work (actually, an additional test-while-you-develop, or build-in-quality step) your project can be actually delivered in 6 months!  

Those are not hypothetical numbers, are real numbers, proved in countless projects actually following agile practices. Actually, this is Agile development. 

Why it is so uncommon to actually automate tests?

Definitely in the Sites world, the biggest obstacle is the lack of tools allowing you to write real tests for your code.  In Sites, JSP does not allow for an easy way to automate tests. You have to resort to tools implementing browser automation like selenium. 

The problem with browser automation is you end up testing the full rendered html page, the whole thing, the final deliverable. Experience shows if have to test the whole thing without splitting in parts, it is very inconvenient since the combination of the possibilities is huge. 

It is much better if you can test the building blocks in isolation. That is what AgileSites test support allows for. Next week I will write a post explaining how AgileSites actually helps you to separate html from java and write unit tests for every parts, but in the meanwhile you can check the relevant page in the tutorial:
http://www.agilesites.org/tutorial/Testing.html

Why separating HTML and Java logic is so important

When I first developed AgileSites, I tried to address a common issue: the endless loop of fix-the-html-when-the-agency-updates it. I solved the problem but I was unaware of the scale such a problem can impact real-word projects.

What is the problem with JSP?

In the ideal world in the mind of people who designed JSP, HTML designer are so smart to be able to directly edit the JSP when they design the HTML.  Better: HTML designer should be so smart to figure out exactly which template they should edit (knowing the content model perfectly) and change it directly in the CMS, then publish it and the site is updates

In the real world, HTML designers have no clue how the CMS work, and no idea of the underlying content model. Better: they don't even want to know: their job is complete once they delivered the mockup of the HTML. It is the JSP coders job to implement it in the CMS

Also in the ideal world of Web Agencies, the HTML is done only once, and it is perfect. It completely satisfies the requirements, it works with all the browsers and the mobile devices in the world, so it can be handed over only once to JSP developers for implementation in the CMS.

In the real world, the HTML is a never finished work. It is first delivered incomplete, then it is updated more or less one thousand of times before it is complete. In the meanwhile the customer changes his mind on the average 2 times a day, and the HTML designers keep updating the HTML mockup.

In the middle, JSP developers go crazy cutting and pasting and diffing the changes made to the HTML, they struggle fixing the JSP to keep up with HTML changes, also because there is a lot of rendering logic embedded in the HTML.

What is the AgileSites solution?

Because I had a first-hand experience of this nightmare, I designed AgileSites in order to keep the HTML totally separated from the Java code,  allowing for totally independent editing of Java and HTML.

The basic idea is pretty simple: you keep the HTML in its original form. You only ask HTML designers to clearly mark where the content should go using normal HTML conventions (usually ids and classes).

When the HTML is delivered, Java coders simply write Java code that will load the HTML in its original form, cut it in smaller pieces (and this cut work is done in code, not manually) then replace the HTML with the content taken from the HTML. That is all. This is what AgileSites allows you to do. This reason alone is enough to use it.

The great benefit is that now HTML editors can change the HTML at their hearth content. As long as they keep the ids they defined in the first place (and usually this is something they do because an id is a convenient placeholder for something they need to keep track of) the HTML can be completely restructured, the look can change completely.

All of this without impacting the Java code that is already written.  Java code must be changed only when new content is needed, not when the look of old content changes.

Some numbers

When I first designed this system, I was unaware of the scale this method can impact the development process of a large site. In a large project, done with just Sites and JSP, the HTML was changing all the time.  I would say that is normal

In that project, there were just 3 HTML developers, but for implementing that in the CMS 5 JSP coders were needed, busy all the time changing the JSP and writing tons of code to keep up with the pace of the HTML developers. The time to implement a change was one week at the minimum for simple changes, to became 2 to 3 weeks when the HTML changes were more complex.

I am now working in a much larger project, very rich in HTML and Javascript coding. Luckily for them, they adopted AgileSites.  Numbers here are very different: there are at least 50 HTML/Javascript developer producing HTML changes at a very high rate. Applying the numbers I saw in the preceding project for keeping the JSP updated you would need maybe 70 JSP coders!

But thanks the AgileSites approach, only TWO Java developer (yes TWO) are required to do some occasional change, that is usually done in a matter of minutes!

AgileSites 1.8 has been released

A short post for a long work. AgileSites 1.8 has been released.

Go to the new website: www.agilesites.org