LDCX Evaluation for ESWC 2014 Demo

On this page, we describe an evaluation not considering drill-across of the Linked Data Cubes Explorer (LDCX).

We want to show that we successfully applied the Linked Data Cubes Explorer (LDCX). For that, we first describe a user study to evaluate the usability and use- fulness of LDCX. Then we describe a performance evaluation to gain an understanding of possible performance bottlenecks of LDCX.

To confirm the usefulness of LDCX and olap4ld, we have put LDCX online and have asked 20 business engineering students at KIT to fill in an online questionnaire ( see at [1] and below) that is inspired by the Computer System Usability Questionnaire (CSUQ) (James R. Lewis. IBM computer usability satisfaction questionnaires: psychometric evaluation and instructions for use. Int. J. Hum.-Comput. Interact., 1995.). With part 1, a set of tasks that students had to solve using the system, we want to investigate whether participants can successfully solve tasks and ensure that participants use all functionality of LDCX. With part 2, a set of statements for which students had to indicate how strongly they agree or disagree, we intend to assess the usability of LDCX and identify potential usability issues.

We have only used 11 -- from 19 of the original CSUQ -- questions most relevant to our system. For instance, we felt the results from the question "The interface of this system is pleasant." already be covered by other questions and the question "The information is effective in helping me complete my work." to not fit the evaluation's context of having students complete tasks that they normally would not execute. Having a reduced amount of questions and participants not experienced or familiar with the type of tasks is a limitation to our evaluation; still, as a tool targeting citizens for exploring Governmental Statistics, we expect the results give an impression of the usability and usefulness of LDCX.

User Study

Part 1: TASKS

Before making LDCX as well as the questionnaire available for the wider public (e.g., announcing the demo at the GLD Working Group and the Publishing Statistical Data Google Group) 8 students have taken part in the evaluation. According to the results of the first part of the evaluation, overall, students were able to fulfil the tasks. All students were able to understand the three-step interface and the visualisation as pivot tables (Task 1, 2, 3, 4, 8); selecting datasets (Task 5, 8, 9, 10); projection (Task 1, 7); and slice (Task 5, 6, 7). For Task 8, 10, 11, two students missed to further drill-down to specific stock market and balance sheet values which probably was because the respective dimensions were not labelled intuitively. For the more complex Task 10 and 11, three students did not wait long enough for the answers. Only one participant could name the failed integrity constraint in Task 9, all others only noticed the general error message. This was expected since those students were not involved in publishing QB datasets.

Part 2: FEEDBACK

The second part of the user study evaluation gave a more detailed idea of how usable the students found LDCX: We computed an overall CSUQ score by averaging over all answers. An overall score of 2.5 shows that students were satisfied with LDCX. Students mentioned that they understood the three-step interface within minutes and found the system easy to use. Worst scores are given in questions related to error messages (Feedback 6) and performance of the system (Feedback 3 and 10). Also, apparently, the search for datasets can be improved: students sometimes chose the wrong dataset (Task 9) and complained about the copy-paste needed to select a dataset. In summary, the system seems usable and robust; improvements are possible regarding the performance, error messages and selection of datasets.

Questionnaire

Website: https://docs.google.com/forms/d/1J721_e_eCCHjghLny1ZPoeb4X5g2h3RAetK7te_Vw6I/viewform

-----------------------
Thank you for helping us improve the Linked Data Cubes Explorer (LDCX), a new analysis tool for statistical data.
Please return your answers by simply filling in this form. You can use English or German.

BACKGROUND INFO
------------------------------------------
What is it about? We want to have your opinion about a new analysis tool for statistical data: Linked Data Cubes Explorer (LDCX). Therefore, we would like you to use our tool for some statistical data analysis tasks and to tell us your experiences.

What do you need to participate? You need an internet connection and a Firefox or Chrome browser.

What do you have to do? There are a couple of tasks to solve. Please enter your answers right after each question. You can also give a comment. Afterwards, you can give feedback about the system.

How long will it take? Solving the tasks should not take longer than 30 minutes, probably you will finish much earlier. Giving feedback may take 3 minutes but you can take all the time you want.

Until when are your answers needed? The earlier, the better. We would like to open up the system to the public as soon as possible. However, if you do not find the time to spend 30 minutes in the next days, no problem. Hand in your answers any time you like.

What happens with your answers? Your responses help us improve our LDCX system. All answers and responses will be handled confidentially and anonymously at all times.

START
------------------------------------------
Please open the following website in your Firefox or Chrome browser and fill in the tasks and feedback statements on the next pages of this form:

http://www.ldcx.linked-data-cubes.org:8000/ldcx-trunk/ldcx/ld-cubes-explorer.html

TASKS

------------------------------------------

Please do the following tasks in turn. Simply state your answer below a task:

Task 1) Name two measures that the "Example SSB dataset" contains.

Task 2) What overall revenue was made by "Customer 2" in the "Example SSB dataset"?

Task 3) Has there been any revenue from Product "Part 2" in the "Example SSB dataset"?

Task 4) From what Date is the revenue of Product "Part 1" in the "Example SSB dataset"?

Task 5) What was the average GDP per capita in PPS for Germany?

Task 6) What was the average GDP per capita in PPS for Germany in 2012?

Task 7) How many values does the dataset for GDP per capita in PPS have for 2012?

Task 8) What was the average Stock Market "Adjusted Closing Price" for Bank of America Corporation on 12 December 2012?

Task 9) What does the system tell you about dataset "HCO3 climate data at location AD0514"? (Hint: The dataset is not modelled correctly)

Task 10) What is the overall average "Available for sale securities noncurrent" in Balance Sheet for COSTCO WHOLESALE CORP, published on 2010-08-29?

Task 11) What were the "Available for sale securities gross realized losses" between 2009-08-31 and 2010-08-29 for COSTCO WHOLESALE CORP?

FEEDBACK

------------------------------------------

Please give your opinions on the following feedback statements:

Simply name the number.

Whenever it is appropriate, please write comments to explain your answers.

Feedback 1) Overall, I am satisfied with how easy it is to use this system.