Skip to main content Link Menu Expand (external link) Document Search Copy Copied

Second Round MFP Patient User Testing Results

Standard Usability Test Findings

                     
Task: Retrieving a prescription from the locker box                  
Role: Patient                  
SUS Question Number 1 2 3 4 5 6 7 8 9 10
Trial I think that I would like to use this system frequently. I found the system unnecessarily complex. I thought the system was easy to use. I think that I would need the support of a technical person to be able to use this system. I found the various functions in this system were well integrated. I thought there was too much inconsistency in this system. I would imagine that most people would learn to use this system very quickly. I found the system very cumbersome to use. I felt very confident using the system. I needed to learn a lot of things before I could get going with this system.
1 5 2 4 1 5 1 5 1 4 2
2 2 1 5 2 3 1 4 1 4 1
3 5 1 5 1 5 1 5 1 5 1
4 4 1 5 1 4 1 5 1 5 1
5 2 1 4 2 4 1 5 2 5 1
6 4 1 5 2 5 1 4 1 4 1
7 5 1 4 2 5 1 4 2 4 1
8 4 2 5 2 4 2 4 1 4 1
9 5 1 5 1 3 3 5 1 3 1
10 4 2 4 1 4 1 5 3 4 1
Data Transposed                    
Trial 1 2 3 4 5 6 7 8 9 10
I think that I would like to use this system frequently. 5 2 5 4 2 4 5 4 5 4
I found the system unnecessarily complex. 2 1 1 1 1 1 1 2 1 2
I thought the system was easy to use. 4 5 5 5 4 5 4 5 5 4
I think that I would need the support of a technical person to be able to use this system. 1 2 1 1 2 2 2 2 1 1
I found the various functions in this system were well integrated. 5 3 5 4 4 5 5 4 3 4
I thought there was too much inconsistency in this system. 1 1 1 1 1 1 1 2 3 1
I would imagine that most people would learn to use this system very quickly. 5 4 5 5 5 4 4 4 5 5
I found the system very cumbersome to use. 1 1 1 1 2 1 2 1 1 3
I felt very confident using the system. 4 4 5 5 5 4 4 4 3 4
I needed to learn a lot of things before I could get going with this system. 2 1 1 1 1 1 1 1 1 1
Data Converted                    
Trial/SUS Question 1 2 3 4 5 6 7 8 9 10
1 4 1 4 3 1 3 4 3 4 3
2 3 4 4 4 4 4 4 3 4 3
3 3 4 4 4 3 4 3 4 4 3
4 4 3 4 4 3 3 3 3 4 4
5 4 2 4 3 3 4 4 3 2 3
6 4 4 4 4 4 4 4 3 2 4
7 4 3 4 4 4 3 3 3 4 4
8 4 4 4 4 3 4 3 4 4 2
9 3 3 4 4 4 3 3 3 2 3
10 3 4 4 4 4 4 4 4 4 4
Converted (STANDARD) 90 80 100 95 82.5 90 87.5 82.5 85 82.5
  Standard Format Scoring Usable Subscale Scoring Learnability Subscale Scoring              
Number of Samples 10 10 10              
Mean SUS Score 87.5 86.25 92.5              
Standard Deviation SUS Score 6.35 7.40 6.45              
Degrees of Freedom 9 9 9              
Confidence Level 95 95 95              
alpha 0.05 0.05 0.05              
t-value 2.26 2.26 2.26              
Margin of Error 4.54 5.29 4.62              
Min SUS Conf. Interval 82.96 80.96 87.88              
Max SUS Conf Interval 92.04 91.54 97.12              
Mean LTR 8 8 9              
Min LTR 8 8 8              
Max LTR 9 9 9              
NPS Status from Mean LTR NEUTRAL NEUTRAL PROMOTER              
Mean Grade A+ A+ A+              
Min Grade A A A+              
Max Grade A+ A+ A+              

Task Completion Times and Success Rates

                                         
Trial 1   2   3   4   5   6   7   8   9   10  
Procedure Steps: Success (1=Yes,0=No) Time (seconds) Success (1=Yes,0=No) Time (seconds) Success (1=Yes,0=No) Time (seconds) Success (1=Yes,0=No) Time (seconds) Success (1=Yes,0=No) Time (seconds) Success (1=Yes,0=No) Time (seconds) Success (1=Yes,0=No) Time (seconds) Success (1=Yes,0=No) Time (seconds) Success (1=Yes,0=No) Time (seconds) Success (1=Yes,0=No) Time (seconds)
The participant is informed by the observer that they will be instructed to pick up a prescription 1 2 1 2.4 1 2 1 1 1 1 1 1 1 1 1 0 1 1 1 0
The patient receives a text message 1 2 1 10 1 0.5 1 17.77 1 20.92 1 11.21 1 18.89 1 12.9 1 7.15 1 13.75
The patient moves over to the box 1 2 1 1 1 17.61 1 1.3 1 6.42 1 6.16 1 2.51 1 3.81 1 1 1 1.06
The patient enters code into the keypad successfully 0 27 1 10 1 11.53 1 7.23 1 29.46 1 5.04 1 3.55 1 9.86 0 21.47 1 18.22
The patient opens the appropriate box 1 1.2 1 1 1 0.71 1 1.85 1 0.88 1 3.61 1 4.95 1 4.89 1 4.16 1 6.75
The patient retrieves the prescription from the box 1 5 0 10 1 7.64 1 5.73 1 3.13 1 4.28 1 5.47 1 2.84 1 2.36 1 0.6
The patient closes the box 1 2 0 0 1 0.5 1 2.25 1 7.94 1 2.55 1 0.24 1 2.51 1 3.8 1 1.3
The Patient is aware that the task is over 1 1 0 0   0.5 1 1 1 1 1 1 1 0.1 1 0 1 0 1 1
Total   42.2   34.4   40.99   38.13   70.75   34.85   36.71   36.81   40.94   42.68
                                         
Completion Rate                                        
Trial/Task 1 2 3 4 5 6 7 8 9 10 Success Rate                  
1 1 1 1 1 1 1 1 1 1 1 1                  
2 1 1 1 1 1 1 1 1 1 1 1                  
3 1 1 1 1 1 1 1 1 1 1 1                  
4 0 1 1 1 1 1 1 1 0 1 0.8                  
5 1 1 1 1 1 0 0 1 1 1 0.8                  
6 1 0 1 1 1 0 0 1 1 1 0.7                  
7 1 0 1 1 1 1 1 1 1 1 0.9                  
8 1 0   1 1 1 1 1 1 1 0.889                  
                                         
Task Times                                        
Time/Task 1 2 3 4 5 6 7 8 9 10 Mean Task Time                  
1 2 2.4 2 1 1 1 1 0 1 0 1.14                  
2 2 10 0.5 17.77 20.92 20.92 18.89 12.9 7.15 13.75 12.48                  
3 2 1 17.61 1.3 6.42 6.42 2.51 3.81 1 1.06 4.313                  
4 27 10 11.53 7.23 29.46 29.46 3.55 9.86 21.47 18.22 16.778                  
5 1.2 1 0.71 1.85 0.88 0.88 4.95 4.89 4.16 6.75 2.727                  
6 5 10 7.64 5.73 3.13 3.13 5.47 2.84 2.36 0.6 4.59                  
7 2 0 0.5 2.25 7.94 7.94 0.24 2.51 3.8 1.3 2.848                  
8 1 0 0.5 1 1 1 0.1 0 0 1 0.56                  
                                         
Total Trial Time 43.2 36.4 43.99 42.13 75.75 76.75 43.71 44.81 49.94 52.68 50.936                  
                                         
Data Analysis                                        
Success Rates Task (Step)                                      
User 1 2 3 4 5 6 7 8                        
1 1 1 1 0 1 1 1 1                        
2 1 1 1 1 1 0 0 0                        
3 1 1 1 1 1 1 1                          
4 1 1 1 1 1 1 1 1                        
5 1 1 1 1 1 1 1 1                        
6 1 1 1 1 0 0 1 1                        
7 1 1 1 1 0 0 1 1                        
8 1 1 1 1 1 1 1 1                        
9 1 1 1 0 1 1 1 1                        
10 1 1 1 1 1 1 1 1                        
Success Rate 1 1 1 0.8 0.8 0.7 0.9 0.889                        
Mean 1 1.091 1.182 1.091 1.182 1.182 1.455 1.6                        
Standard Deviation 0 0.302 0.603 1.044 1.328 1.662 1.864 2.271                        
Number of Samples 10 10 10 10 10 10 10 9                        
Number of Success 10 10 10 8 8 7 9 8                        
Number of Failures 0 0 0 2 2 3 1 1                        
Confidence Level 95 95 95 95 95 95 95 95                        
alpha 0.05 0.05 0.05 0.05 0.05 0.05 0.05 0.05                        
z-critical 1.959963986 1.959963986 1.959963986 1.959963986 1.959963986 1.959963986 1.959963986 1.959963986                        
numerator p_adj 11.92 11.92 11.92 9.92 9.92 8.92 10.92 9.92                        
denominator p_adj 13.84 13.84 13.84 13.84 13.84 13.84 13.84 12.84                        
p_adj 0.8612716763 0.8612716763 0.8612716763 0.7167630058 0.7167630058 0.6445086705 0.789017341 0.7725856698                        
Margin of Error 0.1821095404 0.1821095404 0.1821095404 0.2373792896 0.2373792896 0.2521790096 0.2149545791 0.2292703097                        
Confidence Interval High 1.043381217 1.043381217 1.043381217 0.9541422954 0.9541422954 0.8966876801 1.00397192 1.001855979                        
Confidence Interval Low 0.6791621359 0.6791621359 0.6791621359 0.4793837162 0.4793837162 0.3923296609 0.574062762 0.5433153601                        
Benchmark 0.7 0.7 0.7 0.7 0.7 0.7 0.7 0.7                        
Best Point Estimate 1 1 1 0.8 0.8 0.7 0.9 0.889                        
Exact Binomial p-value 0.0282475249 0.0282475249 0.0282475249 0.3827827864 0.3827827864 0.6496107184 0.1493083459 0.196003234                        
The Probability that the observed proportion of successes comes from a population greater than the benchmark 97.18% 97.18% 97.18% 61.72% 61.72% 35.04% 85.07% 80.40%                        
Null Hypothesis REJECT REJECT REJECT RETAIN RETAIN RETAIN RETAIN RETAIN                        
                                         
Task Times Task (Step)                                      
User 1 2 3 4 5 6 7 8 Total Task Time                      
1 2 2 2 27 1.2 5 2 1 42.2                      
2 2.4 10 1 10 1 10 0 0 34.4                      
3 2 0.5 17.61 11.53 0.71 7.64 0.5 0.5 40.99                      
4 1 17.77 1.3 7.23 1.85 5.73 2.25 1 38.13                      
5 1 20.92 6.42 29.46 0.88 3.13 7.94 1 70.75                      
6 1 20.92 6.42 29.46 0.88 3.13 7.94 1 70.75                      
7 1 18.89 2.51 3.55 4.95 5.47 0.24 0.1 36.71                      
8 0 12.9 3.81 9.86 4.89 2.84 2.51 0 36.81                      
9 1 7.15 1 21.47 4.16 2.36 3.8 0 40.94                      
10 0 13.75 1.06 18.22 6.75 0.6 1.3 1 42.68                      
User/Task Time Task Time Log Task Time                                    
1 42.2 3.742420221                                    
2 34.4 3.538056564                                    
3 40.99 3.713328135                                    
4 38.13 3.641001374                                    
5 70.75 4.259152537                                    
6 70.75 4.259152537                                    
7 36.71 3.603049198                                    
8 36.81 3.605769547                                    
9 40.94 3.71210758                                    
10 42.68 3.753730426                                    
  Arithmetic Geometric Geometric Converted                                  
Mean 45.436 3.782776812 43.93787972                                  
Median 40.965 3.712717857 40.96499237                                  
Number of Samples 10 10 22026.46579                                  
Standard Deviation 13.60297043 0.260294519 1.297312113                                  
Standard Error of the Mean (SEM) 4.301636949 0.08231235424 1.08579491                                  
Confidence Level 95 95 95                                  
alpha 0.05 0.05 0.05                                  
Critical t-value for alpha two-tail 2.262157163 2.262157163 9.603783763                                  
Critical t-value for alpha one-tail 1.833112933 1.833112933 6.253322564                                  
Margin of Error two-tail 9.730978837 0.1862034817 1.204667363                                  
Margin of Error one-tail 7.885386324 0.1508878411 1.162866225                                  
Confidence interval two-tail (high) 47.69815716 6.044933975 421.9698958                                  
Confidence interval two-tail (low) 43.17384284 1.520619649 4.575059248                                  
Confidence interval one-tail (high) 53.32138632 3.933664653 51.09387632                                  
Confidence interval one-tail (low) 37.55061368 3.631888971 37.78412235                                  
Benchmark 60 4.094344562 60                                  
Observed Difference 14.564 0.3115677504 1.365564301                                  
t-value 3.385687861 3.78518818 44.04395799                                  
p-value two-tail 0.008053984486 0.004314451985 1.004323773                                  
p-value one-tail 0.004026992243 0.002157225993 1.002159554                                  
Null Hypothesis one-tail REJECT REJECT RETAIN                                  
Null Hypothesis two-tail REJECT REJECT RETAIN                                 Â