Business forms and process automation using SharePoint 2010

Author: Slava Gorbunov
Email: slava@wyldesolutions.com
Company: Wylde Solutions
Sydney, Australia
February 2011

© 2011 Wylde Solutions. All rights reserved. It is forbidden to reproduce any part of this document, in any form, without prior written permission from the author.

Table of contents:

Prelude
1. Scenario Overview
2. Incident Form Design and Implementation
2.1. Option 1 - SharePoint Form Library
2.2. Option 2 - Infopath Form for External List
2.3. Option 3 - Infopath Form for Custom List
2.4. Decision Making - Form Implementation Approach
3. Incident Management Workflow Design and Implementation
4. Extra Work Required
Conclusions

Prelude

Process automation and implementation of forms is an important part of any business collaboration. Business process automation will increase efficiency and reduce costs and will also help organise the work of a business unit or a team or a single person within the organisation.

With SharePoint 2010 we have plenty of features that allow us to design and implement business workflows and business forms to take advantage of new technologies. When we use technology properly we benefit from having efficient business processes and thus better leveraging of our resources to achieve better results for the business.

This publication is a walk-through of the business process design and implementation using SharePoint 2010, InfoPath 2010 and Visio 2010.

We will use a hypothetical Incident Management Workflow to showcase the process of:



We will discuss different available approaches, advantages and disadvantages and also reasoning behind decision making for each choice.

We will not discuss third-party workflow and form design and management software, because the concept of this publication is to demonstrate how we could effectively use the tools we have got already: SharePoint 2010, Office 2010 and Visual Studio 2010 to be able to create business solutions.
  
In this work we use the versions of software as of January 2011. There is a SharePoint 2010 farm with a database server (Windows 2008 Server R2, SQL Server 2008 R2) and a Web server (Windows 2008 Server R2, SharePoint Server 2010 Enterprise).

1. Scenario Overview

Many organizations implement Incident Management solutions in different ways. This is requirement, and sometimes legal requirement to process incidents, investigate them, report to police or medical service depending on incident severity and provide outcomes if required.

The high-level requirements might be as follows:
We have used the following process for our demonstration:

    1. Operator submits an incident into the system
    2. Business Unit Manager is notified and then investigates an incident and provide extra details.
    Business Unit Manager can also update severity of the incident.
    3. If severity is high then Operations Manager is notified
    3.1. if there is not enough information or some details are missing then Operations Manager returns the incident back to Business Unit Manager
    3.2. if the information is alright then provide the outcomes and report information to external entities if required

Process diagram is shown below:

Incident Management Process

You can download PDF version of the process design.

We decided to use InfoPath Designer 2010 as an incident form designer and we are going to implement browser-enabled form so our users will be able to submit incidents using just browser. That would allow us to improve our process and will give users an option to submit information virtually anywhere. Also with leveraging InfoPath 2010 we are aiming to allow our business users and/or business process owners or analysts to be able to create or modify form design and thus to avoid costs that relate to custom coding as much as possible.

We also decided to use Visio 2010 Premium and SharePoint Designer 2010 to design and implement Incident Management Workflow. That will show us how we can use the Microsoft tools effectively to built business solution for our requirements. We will use these tools as much as possible and will see if we require Visual Studio 2010 when we don't have enough options with complete no-coding approach.

Next step will be Incident Form design and implementation.

2. Incident Form Design and Implementation

First we will have a look at the options we have to implement our Incidents Repository and how we will create an Incident Form. Then we will select the right option for us.

2.1. Option 1 - SharePoint Form Library

Form Library is a type of document library that has InfoPath Form as a document template:

Form Library

We will create a Form Library called Incidents Forms and will use InfoPath Designer 2010 to design form template for Incident.

2.1.1. Design InfoPath Form Template

2.1.1.1. Start InfoPath Designer 2010 and select New -> SharePoint Form Library in available form templates (Popular Form Templates). Then click Design Form button.



2.1.1.2. Add necessary main data source fields:



2.1.1.2.1. In regard to BUManager (Business Unit Manager) and OperationsManager - we are going to use those people further in the workflow so we used a Person/Group Picker control from Controls tab on the ribbon to create those data source fields:

Controls Tab

People Picker Control

2.1.1.2.2. When we add this control to the form InfoPath creates a field group for us:

Field group

2.1.1.2.3. We will rename it to whatever we need:

Rename

BU Manager

2.1.1.2.4. Will do the same for other people in our form (in this scenario Operations Manager).

2.1.1.3. Add extra data connections

In our scenario we will use some additional information, like Incident Source Types and Incident Types. We will store this data in SharePoint lists so users can easily create new or edit existing when necessary.
With browser-enabled forms we have to create a Data Connection for the form and a data connection file and store it in SharePoint in order to access external data on the form.

2.1.1.3.1. Create a new Data Connection

2.1.1.3.1.1. On the Data tab select Data Connections button:

Data Connections button

2.1.1.3.1.2. Then press Add to add new connection:

Add new connection 1

2.1.1.3.1.3. Check "Receive data" check box and then press Next
 
New connection type

2.1.1.3.1.4. Select a "SharePoint library or list" option and press Next



2.1.1.3.1.5. Specify the SharePoint site where this data sits:



2.1.1.3.1.6. Select a list (Incident Source Types in our case):



2.1.1.3.1.7. Select necessary columns:



2.1.1.3.1.8. We don't want to store a copy of the data in the form template so we just leave the checkbox blank and click Next:



2.1.1.3.1.9. Finish the data source creation:



2.1.1.3.2. Do the same for any extra data sources that you require. In our case we will also add Incident Types.

2.1.1.3.3. After we added all data connections, we have to create connection files (UDCX files) to store connection details and to be able to use our data connections in browser-enabled forms that we try to build. If we don't use UDCX files then our forms will not work.
So we have to convert all our data connections to the UDCX files and store them on the SharePoint site in a Data Connection Library.

2.1.1.3.3.1. First we have to create the document library of type Data Connection library on the SharePoint site where we will store incidents, run workflows etc.
So from the Site Actions or from View All Site Content we will create a Data Connection library called Data Connections:




2.1.1.3.3.2. Return to the InfoPath Designer, go to the Data Connections dialog, select the data connection you want to convert and press "Convert to Connection File..." button:



2.1.1.3.3.3. Provide the URL of the data connection library we created before and then add the connection file name so you could find this connection file later by name. In our case the URL we have to provide is "http://site_URL/Data%20Connections/Incident Source Types.udcx":



2.1.1.3.3.4. Do the same for all other data connections.

2.1.1.3.3.5. Go back to Data Connections library, locate the newly created connection file(s) and approve them, because otherwise you will only be able to use those data connections if you have full control on the site:





2.1.1.4. Design form views

When the form is getting too complex in terms of amount of data that we have to display and/or implementing a workflow activities within the form, we need to split the form into several views.
That would allow us to set up different data representations for different stages of the form lifecycle and for different viewers.

2.1.1.4.1. Initial Incident View

This view will be the default one when we create a new incident in our system. We will display only the information that is necessary for submitting an incident and display the rest in other view(s).

Initial Info View

2.1.1.4.2. To create additional view go to Page Design tab and click New View button:



2.1.1.4.3. In our case we wanted to create an additional view for BU Manager. This view would serve as a snapshot of the initial incident data which mostly shouldn't be changed during the process, but only allow to change Severity if required, provide Investigation Comments and select an Operations Manager:



By implementing additional view we simplify the process and give opportunity for users to review data that has been initially provided and only add necessary bits on the next steps within the process. For example, in our case we don't have to create an additional task for Business Unit Managers to complete with separate task record and separate task form, but we can just send a link to the created incident and Business Unit Managers will be able perform their activities with the same form.

2.1.1.4.4. Add and design extra views if required. In our case that might be an Operations Manager view.

2.1.1.5. Create Rules

InfoPath Designer 2010 includes very powerful rules manager. Rules are activities that allow us to implement business logic on the form. For example, validation of different sorts, formatting and actions based on some conditions or not.
In our case we need two simple rules:
  • On form load we need to switch the view based on current Incident Status
  • If Severity is high (Severity equals 1 or 2) then we display the Operations Manager selector for Business Unit Manager, otherwise hide the selector
2.1.1.5.1. Create a rule that works when form loads

2.1.1.5.1.1. Go to the Data tab and select Form Load button in Rules section:



2.1.1.5.1.2. In the New menu on the Rules Manager window select Action item:



2.1.1.5.1.3. We add a condition to this rule:



2.1.1.5.1.4. If Status = "BU":



2.1.1.5.1.5. We need to change the view to BU Manager View. So we add Switch Views action:



2.1.1.5.1.6. Specify the necessary view that we need to switch to if our condition is met on form load:



2.1.1.5.1.7. First rule has been created:



2.1.1.5.2. Rule for the form field value

2.1.1.5.2.1. Select a section of the form that will be affected by the new rule. In our case it is Operations Manager picker section:



2.1.1.5.2.2. Then on the ribbon select Manage Rules:



2.1.1.5.2.3. In the Rules Manager window that will appear, select New -> Formatting item from the menu:



2.1.1.5.2.4. Add condition. In our case we will hide this section if Severity is 3 or 4 or 5:



2.1.1.5.2.5. In the Formatting section check "Hide this control" and we are done with this rule:



2.1.1.5.3. We can add more rules if we like and InfoPath Designer 2010 gives us a very good Rules Manager. If we select a form field or control and then click on Add Rule button on the ribbon we will see the options to create a rule from pre-defined templates:



2.1.1.5.3.1. Available templates are context related so if we select a field of different type it will show us the rule templates that are available for selected field's type. For example, we want to validate Incident Date and it shouldn't be in the future. So we select the Incident Date field and when we press Add Rule button in the ribbon we are able to select from templates that are built for DateTime data fields. So we select the necessary one and it's created in just a click of a button:



2.1.1.5.3.2. Here's the rule we have just created:



2.1.1.6. Specify form submit method

2.1.1.6.1. Select Submit Options button in the File menu:



2.1.1.6.2. Select "Allow users to submit the form" checkbox, "Send form data to a single destination" radio buttion and then select SharePoint document library as a destination. Then click Add for new submit data connection:



2.1.1.6.3. Provide the SharePoint form library URL and specify the file name for the form when it will be saved to that library:



2.1.1.6.4. Give the connection a name and then finish the process:



2.1.2. Publish InfoPath Form Template

After we have designed the form template we need to publish it in order to use. We can publish the form template using three methods:

  • Form Library - to a specified form library where data fields will be promoted to metadata columns and document template for this form library will be the InfoPath Form
  • Site Content Type - form will be published as a content type where data fields will be promoted as Site Columns and document template for this content type will be the InfoPath Form
  • Administrator-approved form template - form template will be prepared for publishing and then must be imported into SharePoint Central Administration for administrators to activate it on certain Site Collections
Form Library option is a basic publishing and is very straightforward, though you will need to publish this form again and again to other similar form libraries if you need to use the same metadata and form template on the other site or in different form library within the same site.
Site Content Type option is more robust, because it allows us to use the same form template and the same set of metadata across the site collection and other site collections if we use Content Type Hub for example. It will require a couple of extra steps to set up after publishing, but it saves time for the future.
Administrator-approved option is the most complex of all three and results are similar to Site Content Type publishing, but require administrative activities to be performed after publishing. Also if your form template contains custom code you will only be allowed to use this option. Other two will be disabled.

In our case we will use Site Content Type publishing method, because we don't want to use custom code at this stage, but will require some flexibility after we publish the form, for example using incident form on sub sites or on the other business units' sites.

2.1.2.1. In the File menu select Publish tab and then SharePoint Server button:



2.1.2.2. Enter the SharePoint site URL:



2.1.2.3. Check the checkbox to Enable this form to be filled by using a browser and select Site Content Type option:



2.1.2.4. Select the option to create or update content type when publishing depending on your circumstances. In our case we will create new:



2.1.2.5. Specify the name and description of the content type:



2.1.2.6. Specify the URL of the form templates library and the file name. Normally there is a document library created by default called Form Templates with the URL as http://siteURL/FormServerTemplates. We will use this document library to store our form template and will name it Incident Form.xsn:



2.1.2.7. Now we have to specify what data fileds from the form will be promoted to Form Library metadata. We will press Add button for a new column:


2.1.2.8. Select a data field and choose a name for the Site Column that will be created as a new or choose existing Site Column to use instead:



2.1.2.9. For columns of type Person or Group in SharePoint there is no analog in InfoPath when you design a form template from scratch. So you cannot directly link your form data column to the SharePoint column of type Person. You can only link one of the data columns representing user in InfoPath to Single line of text column in SharePoint:



2.1.2.10. So when we add all the necessary columns we hit Next and Publish the form template:







2.1.2.11. If we now follow the "Manage this Content Type..." link we will see the details of the Site Content Type that InfoPath Designer created for us:



2.1.2.12. We can't properly use our form template yet, because we have to add the content type we have just created to the Form Library that we will use to submit and store our forms. So in Form Library Settings allow management of content types if you haven't done that yet and add the newly created Site Content Type. In our case it's Incident Form. It will be available under Microsoft InfoPath group:





2.1.2.13. Now we can create a new Incident Form:





2.1.3. Outcomes

This is the first approach we tried in using InfoPath for building business forms for SharePoint 2010. We will overview the benefits and limitations that we faced or will face if we go this path in our scenario.

2.1.3.1. Benefits

Ater we have created the form and published it to the form library we need to understand what are the benefits of using InfoPath in that scenario

2.1.3.1.1. Relatively easy to create and publish functionaly rich browser-enabled forms for using in SharePoint.

2.1.3.1.2. This type of forms - SharePoint Form Library - is very good option if we think about extensibility. We can write custom code to extend the logic if we don't have enough tools in just out-of-the-box InfoPath Designer. This advantage is very valuable when we try to use InfoPath forms in integration with SharePoint and external systems.

2.1.3.1.3. Powerful Rules Manager that allows to implement almost any business logic in regard to validation, formating and working with the form data

2.1.3.2. Limitations

Though InfoPath is a very powerful tool to design business forms there are some limitations that we have to think about when choosing the Form Library template as our form implementation.

2.1.3.2.1. Not all SharePoint column types are supported by Form Library. For example, columns of type Person or Group. We cannot create column of this type when we publish the form and we can't pass the person value into SharePoint without using custom code.
For our case it's very important, because we want to use information about BU Manager and Operations Manager during the workflow, but we can't use just InfoPath Designer 2010 to accomplish this.

2.1.3.2.2. Form attachments will not be indexed by a Search Crawler. For example, if we were to attach an extra document to support incident evidence then this document will be stored within the form as encoded data and we will not be able to search for this document as we could for Custom Lists attachments.

2.2. Option 2 - Infopath Form for External List

In SharePoint 2010 we have a very nice feature - Business Connectivity Services or BCS. This feature allows us to connect SharePoint to external sources of data and easily manage this data in SharePoint by means of External Content Types and External Lists.
So our second option is to try and create InfoPath form for the External List that uses database to store and edit incidents information. We are aiming to have SQL Server database as a central point to manage incidents data so it then be easily accessible by SharePoint users and by reporting software.

Our database model is as follows: Before we start using our incidents data in SharePoint we have to create External Content Types in SharePoint, then create External List for incidents using these External Content Types and then generate an InfoPath form for the External List.

2.2.1. Create External Content Types

2.2.1.1. Open SharePoint Designer 2010, open your SharePoint site, locate the External Content Types tab and then click New Content Type button on the ribbon:



2.2.1.2. You should see the panel for new external content type as below:



2.2.1.3. Type the name for the content type:



2.2.1.4. Click the link near External System selector to select data source:



2.2.1.5. Click Add Connection button:



2.2.1.6. Select SQL Server for Data Source Type:



2.2.1.7. Provide the Database Server and Database Name and depending on your scenario select how would you like to connect to this database:



In our example we will use default value - User's Identity, but if you need more information on how to set up Secure Store Service to provide the way to connect with Impersonated Custom Identity you could have a look at the following presentation for details SharePoint 2010 Integration

2.2.1.8. After you have specified connection settings and got connected to the database you will see the database objects that are available for using in External Content Type. Now we need to generate operations. So we select the certain database table and select Create All Operations from the context menu. That will save us time and will generate all the necessary operations so we could manage this data from SharePoint:



Click Next on the following screen:



Specify columns that you want to display in item picker:



Click Finish and see the operations that were created:



2.2.1.9. Do the same for all tables in the database - create External Content Type for each table that you would like to use. In our case it is Incident Types and Incident Source Types.
2.2.1.10. When all the content types are in the system, create associatons between External Content Types so we could easily build user interface for relationships between them. Select content type from the list and click the Operations Design View button on the ribbon:



2.2.1.11. In the Data Source Explorer select the main table for new association and click New Association in the context menu:



2.2.1.12. Select associated External Content Type. In our case it will be Incident Source Type:





When the content type is selected SharePoint Designer will try to autodetect the field in the main content type that will be linked to the associated content type's primary key:



2.2.1.13. Map the field that will be an identifier for the new association:





2.2.1.14. Finalise the creation process by clicking Finish:



2.2.1.15. Do the same for other External Content Types that have to be associated with the main one. We have added another association for Incident Types in our case.

2.2.1.16. Now we have got all the External Content Types that are necessary to work with our Incidents data in SharePoint:



The next step will be to generate External List and InfoPath form

2.2.2. Create External List and Generate InfoPath Form

2.2.2.1. Open the External Content Type, Incidents in our case and then click Create Lists & Form button on the ribbon:



2.2.2.2. Provide the name for new list and tick the Create InfoPath Form checkbox:



2.2.2.3. The new External List has been created. Now browse to the list details:



2.2.2.4. Click Design Forms in InfoPath -> Item button on the ribbon:



2.2.2.5. The form has been generated:



If we look at the data source fields we will see that InfoPath picked up the mandatory columns and created all the necessary fields that represent our Incidents table and associated data:



Now we have to design the form using similar steps as outlined in 2.1.1. Design InfoPath Form Template starting from 2.1.1.4. Design form views

2.2.3. Outcomes

2.2.3.1. Benefits

2.2.3.1.1. Ability to quickly generate and design rich browser-enabled forms for external data that will be managed via SharePoint.

2.2.3.2. Limitations

2.2.3.2.1. Most significant limitation is that this type of InfoPath forms doesn't allow custom data connections and doesn't allow custom code if we use External Item Pickers - which means we can't use the form properly if we have comprehensive scenarios. For example, if we need to display another data source which is SharePoint list we will not be able to use it and we will not be able to run custom code to get this data in addition to our associations:





2.2.3.2.2. The InfoPath form that is generated can only be used for one certain External List.

2.2.3.2.3. Form modification could be tricky when you update the External Content Type associations. Sometimes it will require complete form creating from scratch which can be costly excercise.

2.2.3.2.4. The forms of that type are not integrated properly with SharePoint. We need to create custom solution which is outside the scope of form design in order to process data from the form into SharePoint 2010.

2.3. Option 3 - Infopath Form for Custom List

In SharePoint 2010 we have another beautiful feature - InfoPath Form for Custom Lists. This feature allows us to modify Custom List item form in InfoPath and then InfoPath will render the list item form to end users in the browser.
So our third option is to try and create InfoPath form for the Custom List called Incidents that we have created to store Incidents data.

2.3.1. Create Content Type and Custom List

2.3.1.1. Create a Site Content Type and add Site Columns. In our case it is a Content Type called Incidents:



2.3.1.2. Create a Custom List, allow management of content types and then add created Content Type to this list:



2.3.2. Create an InfoPath Form

2.3.2.1. From the List Settings got to General Settings -> Form Settings or from the List tab on the ribbon click Customize Form button:



2.3.2.2. InfoPath has generated a form for our list:



It is that easy. Now we have to design the form using similar steps as outlined in 2.1.1. Design InfoPath Form Template starting from 2.1.1.4. Design form views

The only difference will be that we will use the data fields view which is more SharePoint integrated and shows us columns and their types from SharePoint.

2.3.3. Outcomes

2.3.3.1. Benefits

2.3.3.1.1. Ability to quickly generate and design rich browser-enabled forms for Custom Lists in SharePoint.

2.3.3.1.2. Seamless integration with SharePoint and support for almost every column type, including Person or Group. So we don't have to worry about how are we going to pass data to SharePoint from our form so it will be picked up properly.

2.3.3.1.3. Adding columns to/from the form is very easy and straightforward. If we add column in SharePoint we can just refresh our form data source. If we add column on the form then SharePoint will add the column as well:



2.3.3.1.4. Cascading Dropdowns implementation is very easy and doesn't require programming:

2.3.3.1.4.1. Select a dropdown that will depend on the other one. In our case it is Incident Type which depends on the Incident Source Type selection. In the context menu select Drop-Down List Box Properties item:



2.3.3.1.4.2. Near the specified data source for values click Add button:



2.3.3.1.4.3. Leave default values and click Next button:



2.3.3.1.4.4. Select SharePoint library or list option and click Next button:



2.3.3.1.4.5. Specify SharePoint site URL and click Next button:



2.3.3.1.4.6. Select SharePoint list and click Next button. In our case it's Incident Types:



2.3.3.1.4.7. Specify the columns to return and make sure you select the column that links to the parent list. In our case it is Source Type, which is a lookup to Incident Source Types list in SharePoint. Click Next button:



2.3.3.1.4.8. Click Next button:



2.3.3.1.4.9. Specify the new data source name and click Finish button:



We have specified the data source that will return necessary data for us.

2.3.3.1.4.10. Set data source ID column as dropdown Value column:





2.3.3.1.4.11. Select the Entries button:



2.3.3.1.4.12. Click Filter Data button and then click Add button on the filter screen:





2.3.3.1.4.13. Select the column that will be used in filter. Source Type in our case:



2.3.3.1.4.14. Select the filter value as Select field or group:



2.3.3.1.4.15. Select the field in the Main data source. In our case it is Incident Source Type. Then click OK:



2.3.3.1.4.16. Then click OK:



2.3.3.1.4.17. Then click OK:



Now if we change Source Type value then the Incident Type values will be updated in the dropdown.

2.3.3.2. Limitations

2.3.3.2.1. No support for Managed Metadata columns. This one is very disappointing, because a lot of people got very impressed with new features of SharePoint 2010 regarding Managed Metadata, but we can't use InfoPath Form and Managed Metadata together yet:



2.3.3.2.2. Doesn't allow custom code.

2.3.3.2.3. Can only be created for particular list and doesn't support publishing for a Content Type.

2.3.3.2.4. Can only be created for Custom Lists, not Document Libraries.

2.4. Decision Making - Form Implementation Approach

So far we have tried three options of building forms using InfoPath for SharePoint:

We will evaluate these options in compliance with our requirements and process design and taking to account advantages and limitations.

2.4.1. SharePoint is the main system that we want our business users to leverage so we would like to have as much integration of form and SharePoint as possible. So the best choice would be InfoPath Form for Custom List. We don't plan to use Managed Metadata for Incidents so it seems like a good option.

2.4.2. We will not use External List for Incidents and hence the InfoPath form for External List, because we can't run workflows on external data out-of-the-box. In that scenario we will need to develop or purchase custom workflow solution to run and manage workflows using external data. Our requirement is to use as much default functionality as possible.

2.4.3. We want our business users to be able to easily and quickly modify the form if required and they should not need technical experience in order to update the form design and update changes to SharePoint list items. In Form Library template we can't easily work with columns of certain types that are important and supported by InfoPath Form for Custom Lists.

Our choice is Infopath Form for Custom List.