client page that exploit DEEP-Client functionality
DEEP (Dealer Provider)
DEEP-Component
The DEEP-Client ask to DEEP for a specific Datalet, DEEP responds with the information necessary for the Datalet inclusion, the DEEP-Client retrieve the Datalet from DEEP-Component and include it into the page.
A HTML web page that includes webcomponents must include the DEEP-Client Javascript library, configured with the URL of the DEEP repository.
In the above image, a web page contains a data-table webcomponent that is configured with a data_url elements (an URL to a data-source) and a params element (that may contain configuration parameters as requested by the webcomponent).
DEEPCLIENT contacts DEEP and asks for the location of the data-table webcomponent.
DEEP answers with the complete URL where the Client can download the data-table component.
DEEPCLIENT sends a request to the aforementioned webcomponent URL and downloads the data-table webcomponent's code. We say that webcomponents are hosted by a DEEPCOMPONENT repository (basically a web server compatible with HTTP/Ajax that serves webcomponents' HTML code).
Once downloaded the data-table webcomponent's code can be placed in the HTML client page and rendered within the browser.
Code control system
Prerequisites
Web server with PHP support (DEEP is currently developed in PHP)
now in your web server's local directory there are 3 new directory DEEP, DEEPCLIENT, COMPONENTS.
Now we have to configure DEEP and DEEPCLIENT to work correctly on your environment.
In DEEP directory edit file datalets.xml. Substitute the <components_repository_url_reference> value with the full URL of COMPONENTS/datalets just cloned, repeat the same operation for controllets.xml.
In DEEPCLIENT edit file index.html and substitute ComponentService.deep_url with the correct DEEP URL.
Now you can test the Datalet demo pointing the browser to DEEPCLIENT/index.html
Keywords
Web Component
Are a set of standards currently being produced by Google engineers as a W3C specification that allow for the creation of reusable widgets or components in web documents and web applications. Web Components consist of 4 main elements which can be used separately or all together:
Custom Elements
Shadow DOM
HTML Imports
HTML Templates
Polymer
The Polymer library developed by Google is designed to make easier and faster for developers to create reusable components for the modern web.
webcomponents.js polyfills
Are a set of polyfills built on top of the Web Components specifications. It makes possible for developers to use these standards today across all modern browsers. As these technologies are implemented in browsers, the polyfills will shrink and you'll gain the benefits of native implementations. webcomponents.js automatically detects native support and switches to the fast path when available. Your elements seamlessly start relying on the native implementation.
Datalet
It‘s responsible for data retrieving, filtering and visualization. From technological point of view it’s a web component built with polymer library. In our architecture each Datalet inherit from a base web component: base-datalet.
The base-datalet have two main attributes: data_url and query. The data_url attribute specify the data url from which to take data. This data should be json formatted and available by ajax request. The query attribute is a string which represent a selection and, optionally, filtering to apply to retrieved data.
Datalet lifecycle
Each Datalet follow this lifecycle inherit from base-datalet:
Data retrieving through an ajax request
Data selection
Data filtering (optional)
Data transformation
Data presentation
Controllet
It’s a web component created with Polymer library that implements control behaviour like form, slider, etc. It generally use a set of polymer core elements to realize a more complex component using composition paradigm.
DEalEr Provider (DEEP)
The Dealer provider (DEEP) is a RESTful web service written in PHP. It make available a public discovery service that retrieve all the Datalet available into the system and a service call that provide a specific Datalet.
DEEPCLIENT
Is a simple javascript library that allows the connection with DEEP and the inclusion of a Datalet within a HTML page.
We create an empty HTML document with a single DIV. On document ready event we create a datalet_params object, this object contains information for Datalet retrieving and initialization.
datalet_params contains :
component : the Datalet name to be sent to DEEP (Eg. datatable-datalet)
params.data-url : the datasource URL (Eg.http://demo.ckan.org/api/action/datastore_search?resource_id=8324a4c9-66d6-47bf-a898-94237cc39b9f&limit=50)
fields : an array with user selected fields from datasource (Eg. Array("Supplier", "Amount"))
placeHolder : the ID of the DOM element where place the Datalet
ComponentService.getComponent functiontake a datalet_params object as parameter. This function is responsible for Datalet info retrieving from DEPP and Datalet code retrieving from DEEPCOMPONENTS. Once the Datalet code is available DEEPCLIENT add the Datalet to the document DOM and initialize the Datalet with the data and fields parameters.
Now the Datalet is able to complete its lifecycle.
The programmer must specify the DEEP endpoint through the variable ComponentService.deep_url.
Since Polymer 1.0 doesn’t support components inheritance, we provide a way to have it in datalets, waiting for it in the next Polymer version.
We provide a bundle of objects that implements all behaviors we want our datalets have. They are divided in two categories : BaseDataletsBehaviors and CustomDataletBehaviors.
The BaseDataletBehaviors define the lifecycle for all Datalets. Every lifecycle step in BaseDataletBehaviors is implemented in the CustomDataletBehaviors. This mechanism provides a components hierarchy that standardize Datalet production and make reusable common behaviors.
The BaseDataletBehaviors are:
BaseDataletBehavior
WorkcycleBehavior
The BaseDataletBehavior define the mandatory attributes that all datalet must have :
data_url (string) : the URL used to get the data from the Open Data Provider (e.g. the CKAN API).
fields (String): it represents a JSON array of user selected dataset fields.
data (array): data structure that store the data retrieved from Open Data Provider.
varBaseDataletBehavior= {
properties: {
/**
*It represent the data url from CKAN api
*
*@attribute dataUrl
*@typestring
*@default'null'
*/
dataUrl: {
type:String,
value:""
},
/**
*It represents one or multiple fields selected by user
*
*@attribute fields
*@typeString
*@default empty
*/
fields: {
type:String,
value:""
},
/**
*The selected and transformed data you can usein presentation phase
*
*@attribute data
*@typeArray
*@default empty
*/
data: {
type:Array,
value:[]
}
},
factoryImpl:function(data_url, fields) {
this.data_url = data_url;
this.fields = fields;
}
};
The WorkcycleBehavior implements the workcycle of each datalets, composed by this steps :
Get the datafrom a source : It’s responsible for data retrieving from an Open Data Platform. It’s the starting point of Datalet work-cycle.
Select a subset of fields from the data : this operation is data format dependent. It’s responsible for extract a query related subset of information from the entire dataset. An multidimensional array will be made available for the transformation step.
Filter the data by applying logical operation( <, >, =, <=, >=, etc.): this function is designed to apply logical operation to the data created in the selection step (this step and the previous one could be merged together).
Transform data in order to obtain a coherent data rapresentation
Present data the data for a specific visualization : here we create the data structure necessary for visualization libraries initialization. At this point could be an interaction with component’s DOM.
varWorkcycleBehavior= {
_component:null,
/**
*Request data from source(e.g. CKAN by api)using some kind of technology(e.g.Ajax)
*
*@method requestData
*/
requestData:function(){
},
/**
*Select the fields from data(typically json) previously retrieved by ajax request.The selection could be doneby jsonPath but
* it depends on the representation data format(CKAN apies return a json representation of the dataset).
*
*@method selectData
*/
selectData:function(){
},
/**
*Filter data previously selected.An example of filterting could be an expression such "fields > 30"or"fields = 'AAA'"
*If you are using jsonPath to select the datas you can apply an expression directly in the jsonPath query string.
*
*@method filterData
*/
filterData:function(){
},
/**
*Transform the selected data in order to build the structure that the presentation phase needs.
*
*@method transformData
*/
transformData:function(){
},
/**
*Build the object/s for presentation layer.
*
*@method presentData
*/
presentData:function(){
},
/**
*This method represents the entire datalet workcycle.
*
*@method runWorkcycle
*/
runWorkcycle:function() {
this.selectData();
this.filterData();
this.transformData();
this.presentData();
},
/**
*This method save the reference to the polymer object related to the datalet.
*
*@method init
*/
init:function(component){
this._component = component;
this.requestData();
}
};
The CustomDataletBehaviors represents the specific implementation for the work-cycle steps.
This behaviours are a components hierarchy that use a particular communication channel, data format and visualization library.
An example of datalet for Highchart js library
We’ll describe for example purpose a Datalet that use a AJAX channal, handle JSON data format and use Highcharts for data visualization and JSONPATH library for data selection.
Ajax-Json-JsonPath behavior
Here we exploit an AJAX communication channel that handle JSON data format and extends the base behaviors (in particular the requestData method). This behavior override the select method in order to perform a JSONPATH query according to the user selected fields stored in the this._component.fields variable.
varAjaxJsonJsonPathBehavior= {
properties: {
/**
*It contains the json data from async xhr call returned from core-ajax core component
*
*@attribute json_results
*@typeobject
*@default'null'.
*/
json_results: {
type:Object,
value:{}
}
},
/**
*Make an AJAX call to the dataset URL
*
*@method requestData
*/
requestData:function(){
var comp =this;
$.ajax({
url:this._component.dataUrl,
dataType:"json",
success:function(e){
comp.handleResponse(e);
}
});
},
/**
*Calledwhen core-ajax component receive the json data from called url.
*
*@method handleResponse
*/
handleResponse:function(e) {
this.properties.json_results.value = e;
this.runWorkcycle();
},
/**
* selectData built a JSONPATH query based on the user selected fields then extract data from the JSON response.
*This method built an objects <name, data>for every user selected field and push it into the data array.
this.data.push({name : query_elements[query_elements.length -1], data : jsonPath(this.properties.json_results.value, query)});
}
}
};
json_results(object): the object that will store the JSON results returned from data request. It will be used from the Datalets which will use this behavior in order to access to this response object everywhere.
requestData: it’s an override of WorkcicleBehavior component to use AJAX to request data
handleResponse: it’s callback for AJAX request that set the json_response attribute that will be used by the specialized behavior that will use the data (select, filter and transform) and run the workcycle.
Base datalet
We define a base-datalet that include BaseDataletBehaviorand a footer that contain a ROUTE-TO-PA logo and data source domain information.
All components that import this one will include AjaxJsonJsonPathBehavior and JSONPath library. Since this Datalet handle JSON data, we use JSONPath (http://goessner.net/articles/JsonPath/) to select specific fields from entire dataset (an example of query is $.result.records..Supplier $.result.records..Amount). For a new datatype a new behavior and new way to select the fields should be provided.
Highcharts component
The component we’ll present in this section include Highcharts javascript library (http://www.highcharts.com/) to visualize data in a several type of charts.
Since all selected example charts use the same way to select the data to build the categories and series objects, we built a super Highchart component which all our specific chart datalet will “extend” in order to share the selection mechanism.
At this point we include the Highchart javascript library used by all chart datalets which use Highchart as visualization library.
<scriptsrc="static/js/highcharts.js"></script>
<scriptsrc="static/js/exporting.js"></script>
Now we define a new behavior with all attribute that the specialized chart datalets can share (categories and series) and the implementation of a selection method.
//console.log("Parsing data error. Highchart-datalet.selectData");
}
returntrue;
});
this.properties.series.value.push(this.data[i]);
}
}
};
We are overriding the transformData method of WorkcycleBehavior so to access to data attribute we have to use the reference to the Polymer component (this._component). This reference will be set by the final component that implements all workcycle (that visualize the chart in our example), as we will show you later. As you can notice in the transformData function we can access to json_results attribute defined in the Behavior that implements the communication channel.
After defined the Behavior for Highchart component we have to extend it to include the BaseDataletBehavior, WorkcycleBehavior, AjaxJsonJsonPathBehavior to crate a new behavior specific for our new Datalet. The order of the extension is crucial (see JQuery $.extend docs).
The last thing to do is to create the Polymer javascript object.
HighchartsDatalet=Polymer({
is:'highcharts-datalet'
});
Linechart datalet
The linechart-datalet will use the highchart-component to visualize a linechart. To do this we override the presentData method, in a new behavior(LinechartBehavior), in order to use the categories and series attributes, generated by “super class”, to build the chart.
After creating the main component behavior (LinechartComponentBehavior) extending the HighchartComponentBehavior previously created, we have to initialize it by passing the reference to Polymer object associated to linechart-datalet. We do it in the ready handler of Polymer object.
LinechartDatalet=Polymer({
is:'linechart-datalet',
/**
*'ready' callback extend the LinechartComponentBehaviorwithHighchartsComponentBehaviorandLinechartBehavior
The ready handler define a lifecycle callback invoked when all local DOM children of this element have been created and "configured" with data bound from this element, attribute values, or defaults.
The base behavior will use this reference to get information from the Polymer component like passed attributes, DOM, etc.
Barchart datalet
The barchart-datalet will use the highchart-datalet to visualize a barchart.
To be more complete we present another example of datalet that is slightly different from previous ones: a table based datalet which use Datatables javascript library (https://www.datatables.net/) to render a table. The main difference is that datatable-datalet component has a dynamic DOM built according to the size of data.
To document a web component you can use a iron-component-page polymer component(https://elements.polymer-project.org/elements/iron-component-page). Each component must be documented to make DEEP able to give information about available components and their usage.
Documentation can be encoded into html comments (<!-- ... -->) or using JsDoc notation (/** ... */). Markdown format is supported.
Once you have properly commented the code to include the new component (datalet or controllet) to documentation you have to add the component path to source attribute.
Deploy a new component
To deploy a new component we have to :
Upload the new component (html, javascripts and stylesheets) in /COMPONENTS/type-of-component{datalets/controllets}/name-of-component directory. For instance if you have created a my-datalet component you have to copy it in /COMPONENTS/datalets/my-datales/my-datalets-file(html, js, css).
Add a component tag in xml file located in DEEP project: datalets.xml, for a new datalet, controllets.xml, for a new controllet. For instance if you created my-datalet component you have to add to datalets.xml the follow snippet:
<component>
<name>barchart-datalet</name>
<attributes>
<attribute>
<name>data-url</name>
</attribute>
<attribute>
<name>query</name>
</attribute>
</attributes>
<idm>
<inputs>
<input>
<name>Categories</name>
<description>The chart categories. Its values will be put in the horizontal axis. You need one value for each series.</description>
<scale>nominal</scale>
<role>domain</role>
<selection>11</selection>
</input>
<input>
<name>Series</name>
<description>The chart series. Its values will be put in the vertical axis.</description>
<scale>nominal</scale>
<role>domain</role>
<selection>11</selection>
</input>
<layouts>
<input>
<name>x-axis-label</name>
</input>
</layouts>
</inputs>
</idm>
</component>
You should expose all attributes (adding attributes tags) in order to make the DEEP able to give this information when your component will be requested.
MISC
Integrated Development Environment: phpStorm
As an Integrated Development Environment (IDE) for PHP and web component, JetBrains Phpstorm (https://www.jetbrains.com/phpstorm/ ) has been chosen, but any other ide can be used. PhpStorm provides an editor as support for PHP, HTML, Javascript, CSS and many other languages.