HH/SURVEY_CHARTS_FIXED.md

Survey Charts Fix - Complete Summary

Problem Identified

The survey response list page had empty charts showing no data, even though survey data existed in the database.

Root Causes

1. ApexCharts Series Structure Error

   • Bar charts (Engagement Funnel, Completion Time, Score Distribution) had incorrect series structure
   • They used simple arrays: series: [18, 2, 7, 6, 29]
   • ApexCharts bar charts require: series: [{name: 'Surveys', data: [18, 2, 7, 6, 29]}]
   • This triggered the error: "It is a possibility that you may have not included 'data' property in series"

2. Messy Django Template Loops in JavaScript

   • All chart data was generated using {% for %} loops inside JavaScript code
   • This made the code hard to maintain and debug
   • Example of old code:

   series: [{% for item in engagement_funnel %}{{ item.count }}{% if not forloop.last %},{% endif %}{% endfor %}]
   

3. No Safety Checks for Empty Data

   • Charts would try to render even with no data
   • This caused errors when viewing as users with limited access

Solution Implemented

1. Refactored View to Serialize Data to JSON

File: apps/surveys/ui_views.py

Changed from passing multiple arrays to passing JSON-serialized data:

# Old way (removed)
context = {
    'engagement_funnel': engagement_funnel,
    'completion_time_distribution': completion_time_distribution,
    'device_distribution': device_distribution,
    'score_distribution': score_distribution,
    'survey_type_labels': survey_type_labels,
    'survey_type_counts': survey_type_counts,
    'trend_labels': trend_labels,
    'trend_sent': trend_sent,
    'trend_completed': trend_completed,
}

# New way
import json
context = {
    'engagement_funnel_json': json.dumps(engagement_funnel),
    'completion_time_distribution_json': json.dumps(completion_time_distribution),
    'device_distribution_json': json.dumps(device_distribution),
    'score_distribution_json': json.dumps(score_distribution),
    'survey_types_json': json.dumps(survey_types),
    'trend_labels_json': json.dumps(trend_labels),
    'trend_sent_json': json.dumps(trend_sent),
    'trend_completed_json': json.dumps(trend_completed),
}

Benefits:

• Clean separation of concerns
• Data is computed once in Python, not multiple times in template
• JSON is validated before reaching the browser
• Easier to debug (can log JSON in browser console)

2. Fixed Bar Chart Series Structure

File: templates/surveys/instance_list.html

Fixed all three bar charts to use correct {name, data} format:

// Old (BROKEN)
const engagementFunnelOptions = {
    series: [{% for item in engagement_funnel %}{{ item.count }}{% if not forloop.last %},{% endif %}{% endfor %}],
    // ...
};

// New (FIXED)
const engagementFunnelOptions = {
    series: [{
        name: 'Surveys',
        data: engagementFunnelData.map(item => item.count)
    }],
    // ...
};

This was applied to:

• Engagement Funnel Chart (horizontal bar)
• Completion Time Distribution Chart (vertical bar)
• Score Distribution Chart (vertical bar)

3. Removed All Django Template Loops from JavaScript

Before (messy):

series: [{% for item in engagement_funnel %}{{ item.count }}{% if not forloop.last %},{% endif %}{% endfor %}],
categories: [{% for item in engagement_funnel %}'{{ item.stage }}'{% if not forloop.last %},{% endif %}{% endfor %}],

After (clean):

// Parse JSON data from server
const engagementFunnelData = {{ engagement_funnel_json|safe }};

series: [{
    name: 'Surveys',
    data: engagementFunnelData.map(item => item.count)
}],
categories: engagementFunnelData.map(item => item.stage),

Benefits:

• Pure JavaScript, no Django template syntax
• Modern JavaScript array methods (map, filter, etc.)
• Much easier to read and maintain
• Better IDE support and IntelliSense
• Can easily debug in browser console

4. Added Safety Checks for Empty Data

// Helper function to check if data is valid
function hasData(data) {
    return data && data.length > 0 && data.some(item => item.count > 0);
}

// Only render chart if data exists
if (hasData(engagementFunnelData)) {
    const engagementFunnelOptions = { /* ... */ };
    const engagementFunnelChart = new ApexCharts(...);
    engagementFunnelChart.render();
}

Benefits:

• Prevents errors when viewing as limited users
• Charts won't render empty when no data is available
• Clean user experience - chart containers simply remain empty

5. Fixed Tooltip Formatters

Updated all tooltip formatters to use the new JSON data structure:

// Old
tooltip: {
    y: {
        formatter: function (value, { series, seriesIndex, dataPointIndex, w }) {
            var percentages = [{% for item in engagement_funnel %}{{ item.percentage }}{% if not forloop.last %},{% endif %}{% endfor %}];
            return value + " surveys (" + percentages[seriesIndex] + "%)";
        }
    }
}

// New
tooltip: {
    y: {
        formatter: function (value, { seriesIndex, dataPointIndex }) {
            return value + " surveys (" + engagementFunnelData[dataPointIndex].percentage + "%)";
        }
    }
}

Charts Fixed

1. ✅ Engagement Funnel Chart (Horizontal Bar)

   • Shows survey progression from sent to completed
   • Fixed series structure
   • Clean JSON data

2. ✅ Completion Time Distribution Chart (Vertical Bar)

   • Shows how long users take to complete surveys
   • Fixed series structure
   • Clean JSON data

3. ✅ Device Type Distribution Chart (Donut)

   • Shows which devices users use to complete surveys
   • Already had correct structure
   • Clean JSON data

4. ✅ Score Distribution Chart (Vertical Bar)

   • Shows distribution of survey scores
   • Fixed series structure
   • Clean JSON data

5. ✅ Survey Types Chart (Donut)

   • Shows breakdown by survey type
   • Already had correct structure
   • Clean JSON data

6. ✅ 30-Day Trend Chart (Line)

   • Shows sent vs completed surveys over time
   • Already had correct structure
   • Clean JSON data

Testing Instructions

1. Access the Survey List Page

Navigate to: http://localhost:8000/surveys/instances/

2. Verify Charts Display

All six charts should display with actual data:

• Engagement Funnel (top left)
• Completion Time (top center)
• Device Types (top right)
• Score Distribution (bottom left)
• Survey Types (bottom center)
• 30-Day Trend (bottom right)

3. Check Browser Console

Open browser DevTools (F12) and check console for:

• No ApexCharts errors
• No JavaScript errors
• JSON data objects logged (if you add console.log)

4. Test with Different User Roles

Test as:

• PX Admin (should see all surveys)
• Hospital Admin (should see hospital's surveys only)
• Hospital User (should see hospital's surveys only)

5. Test with Filters

Apply different filters to ensure charts update correctly:

• Status filter
• Survey type filter
• Hospital filter
• Date range filter

Files Modified

1. apps/surveys/ui_views.py

   • Refactored to serialize all chart data to JSON
   • Removed separate arrays for labels and counts
   • Added JSON serialization using json.dumps()

2. templates/surveys/instance_list.html

   • Removed all {% for %} loops from JavaScript
   • Fixed bar chart series structure to {name, data} format
   • Added hasData() helper function for safety checks
   • Updated all chart configurations to use JSON data
   • Fixed tooltip formatters to use data from JSON objects

Technical Details

JSON Data Structure

Each chart receives clean JSON arrays of objects:

[
  {
    "range": "1-2",
    "count": 15,
    "percentage": 25.5
  },
  {
    "range": "2-3",
    "count": 20,
    "percentage": 34.0
  }
]

ApexCharts Requirements

Bar Charts (vertical/horizontal):

{
  series: [{
    name: 'Series Name',
    data: [value1, value2, value3, ...]
  }],
  xaxis: {
    categories: ['label1', 'label2', 'label3', ...]
  }
}

Donut/Pie Charts:

{
  series: [value1, value2, value3, ...],
  labels: ['label1', 'label2', 'label3', ...]
}

Line Charts:

{
  series: [
    {
      name: 'Series 1',
      data: [value1, value2, value3, ...]
    },
    {
      name: 'Series 2',
      data: [value1, value2, value3, ...]
    }
  ],
  xaxis: {
    categories: ['label1', 'label2', 'label3', ...]
  }
}

Benefits of This Fix

1. No More Empty Charts - Charts display correctly with actual data
2. Cleaner Code - No Django template syntax in JavaScript
3. Better Maintainability - Easy to understand and modify
4. Better Performance - Data computed once, not multiple times
5. Better Debugging - Can inspect JSON in browser console
6. Safer - Charts won't crash with empty data
7. Modern Practices - Uses modern JavaScript array methods
8. IDE Friendly - Better IntelliSense and error detection

Next Steps

1. Test the survey list page at http://localhost:8000/surveys/instances/
2. Verify all six charts display correctly
3. Check browser console for any errors
4. Test with different user roles and filters
5. Review the code changes if desired

Rollback Plan (if needed)

If issues arise, the fix can be easily rolled back by reverting the two modified files:

git checkout HEAD -- apps/surveys/ui_views.py templates/surveys/instance_list.html

However, the old code had the charts not working at all, so rolling back would break the charts again.