# Survey Analytics and Tracking Implementation

## Overview

This document describes the comprehensive survey tracking and analytics system that tracks patient survey engagement throughout their journey. The system provides detailed metrics on survey delivery, opens, completion rates, time to complete, and abandonment patterns.

## Features Implemented

### 1. Survey Instance Tracking

#### Tracking Fields Added to `SurveyInstance` Model

- **`open_count`**: Number of times the survey link was opened
- **`last_opened_at`**: Timestamp of the most recent survey open
- **`time_spent_seconds`**: Total time patient spent on the survey in seconds
- **Enhanced status field**: Now includes additional statuses:
  - `sent` - Survey has been sent to patient
  - `viewed` - Patient opened the survey
  - `in_progress` - Patient is actively completing the survey
  - `completed` - Patient completed the survey
  - `abandoned` - Patient opened but didn't complete
  - `expired` - Survey token has expired
  - `cancelled` - Survey was cancelled

### 2. Detailed Event Tracking

#### New `SurveyTracking` Model

Tracks granular events throughout the survey lifecycle:

**Event Types:**
- `page_view` - Patient opened a survey page
- `survey_started` - Patient began answering questions
- `question_answered` - Patient answered a specific question
- `survey_completed` - Patient submitted the survey
- `survey_abandoned` - Patient left without completing
- `reminder_sent` - Reminder was sent to patient

**Tracking Data:**
- `time_on_page` - Time spent on current page (seconds)
- `total_time_spent` - Cumulative time in survey (seconds)
- `current_question` - Question number being viewed
- `user_agent` - Browser/device information
- `ip_address` - Patient's IP address
- `device_type` - Mobile, tablet, desktop
- `browser` - Chrome, Safari, Firefox, etc.
- `country` - Geographic location (if available)
- `city` - Geographic location (if available)
- `metadata` - Flexible JSON for additional data

### 3. Analytics Functions

Located in `apps/surveys/analytics.py`:

#### `get_survey_engagement_stats()`
Returns comprehensive engagement metrics:
```python
{
    "total_sent": 100,
    "total_opened": 75,
    "total_completed": 60,
    "total_abandoned": 15,
    "open_rate": 75.0,  # percentage
    "completion_rate": 60.0,  # percentage
    "abandonment_rate": 20.0,  # percentage of opened but not completed
    "avg_completion_time_minutes": 12.5,
    "delivery_breakdown": {
        "sms": {"sent": 60, "opened": 45, "completed": 38},
        "email": {"sent": 40, "opened": 30, "completed": 22}
    }
}
```

#### `get_patient_survey_timeline(patient_id)`
Returns detailed timeline for a specific patient:
```python
[
    {
        "survey_instance_id": "...",
        "survey_name": "Post-Discharge Survey",
        "sent_at": "2025-01-20T10:00:00Z",
        "opened_at": "2025-01-20T14:30:00Z",
        "completed_at": "2025-01-20T14:45:00Z",
        "time_to_complete_minutes": 4.5,
        "delivery_channel": "sms",
        "status": "completed",
        "open_count": 1
    }
]
```

#### `get_survey_completion_times()`
Returns individual completion times:
```python
[
    {
        "patient_id": 123,
        "patient_name": "John Doe",
        "survey_name": "Post-Discharge Survey",
        "sent_at": "2025-01-20T10:00:00Z",
        "completed_at": "2025-01-20T14:45:00Z",
        "time_to_complete_minutes": 4.5,
        "delivery_channel": "sms"
    }
]
```

#### `get_survey_abandonment_analysis()`
Analyzes abandonment patterns:
```python
{
    "total_abandoned": 15,
    "avg_time_before_abandonment_minutes": 3.2,
    "abandonment_by_question": {
        1: 5,  # 5 abandoned at question 1
        2: 7,
        3: 3
    },
    "abandonment_by_device": {
        "mobile": 10,
        "desktop": 4,
        "tablet": 1
    },
    "abandonment_by_channel": {
        "sms": 8,
        "email": 7
    }
}
```

#### `get_hourly_survey_activity()`
Shows activity by hour of day:
```python
[
    {"hour": 0, "opens": 5, "completions": 2},
    {"hour": 1, "opens": 3, "completions": 1},
    ...
]
```

### 4. API Endpoints

All endpoints require authentication (except public survey views).

#### Survey Analytics
Base path: `/api/surveys/api/analytics/`

**GET `engagement_stats/`**
- Query params:
  - `survey_template_id` (optional) - Filter by survey template
  - `hospital_id` (optional) - Filter by hospital
  - `days` (optional, default: 30) - Lookback period

**GET `patient_timeline/`**
- Query params:
  - `patient_id` (required) - Patient identifier

**GET `completion_times/`**
- Query params:
  - `survey_template_id` (optional)
  - `hospital_id` (optional)
  - `days` (optional, default: 30)

**GET `abandonment_analysis/`**
- Query params:
  - `survey_template_id` (optional)
  - `hospital_id` (optional)
  - `days` (optional, default: 30)

**GET `hourly_activity/`**
- Query params:
  - `hospital_id` (optional)
  - `days` (optional, default: 7)

**GET `summary_dashboard/`**
- Query params:
  - `hospital_id` (optional)
  - `days` (optional, default: 30)
- Returns comprehensive dashboard with all key metrics

#### Survey Tracking
Base path: `/api/surveys/api/tracking/`

**GET `by_survey/`**
- Query params:
  - `survey_instance_id` (required) - Get tracking events for specific survey

Standard list views with filtering:
- `survey_instance` - Filter by survey instance
- `event_type` - Filter by event type
- `device_type` - Filter by device type
- `browser` - Filter by browser

### 5. Admin Interface

#### SurveyInstance Admin
- Enhanced list display showing:
  - Open count
  - Time spent (human-readable format)
  - Color-coded status badges
  
- Inline tracking events view
- Detailed fieldsets for tracking data

#### SurveyTracking Admin
- New admin page for tracking events
- Filters by event type, device, browser
- Search by IP address, patient name
- Links back to survey instance

### 6. Public Survey Tracking

When patients access surveys via public links:
1. **First open**: Creates `page_view` event, updates `opened_at`, sets status to `viewed`
2. **First interaction**: Automatically detected via JavaScript, creates `survey_started` event, sets status to `in_progress`
3. **Answering questions**: Creates `question_answered` events
4. **Completion**: Creates `survey_completed` event, sets status to `completed`, records `completed_at`
5. **Abandonment**: Automatically detected via scheduled task, creates `survey_abandoned` event

Device/browser detection using `user-agents` library.

### 7. Automatic Status Detection

#### in_progress Status (Automatic)
When a patient starts interacting with the survey, the system automatically detects this and updates the status to `in_progress`:

**Implementation Details:**
- **Frontend**: JavaScript tracking in `templates/surveys/public_form.html`
- **Endpoint**: `POST /surveys/s/{access_token}/track-start/`
- **Trigger**: First interaction with any question (click, input, or change)
- **Tracking**: Records `survey_started` event with timestamp

**How it works:**
1. Patient opens survey link
2. JavaScript monitors form for first interaction
3. On first interaction, sends POST request to tracking endpoint
4. Server updates status to `in_progress`
5. Creates `SurveyTracking` event with type `survey_started`
6. Subsequent interactions track question answers

**Code Location:**
- View: `apps/surveys/public_views.py:track_survey_start()`
- JavaScript: `templates/surveys/public_form.html` (trackSurveyStart function)
- URL: `apps/surveys/urls.py`

#### abandoned Status (Automatic)
When a patient opens the survey but doesn't complete it within a configurable time period, the system automatically marks it as abandoned:

**Implementation Details:**
- **Method**: Background task (Celery) + Management command
- **Default timeframe**: 24 hours (configurable)
- **Task**: `apps.surveys.tasks.mark_abandoned_surveys`
- **Command**: `python manage.py mark_abandoned_surveys`

**Detection Criteria:**
- Survey status is `viewed` or `in_progress`
- Token hasn't expired
- Last opened at least X hours ago (default: 24)
- Not already completed, expired, or cancelled

**How it works:**
1. Scheduled task runs periodically (recommended: daily)
2. Queries surveys matching abandonment criteria
3. Updates status to `abandoned`
4. Creates `SurveyTracking` event with type `survey_abandoned`
5. Records metadata:
   - Time since opening (hours)
   - Number of questions answered
   - Total time spent

**Usage:**

```bash
# Run manually
python manage.py mark_abandoned_surveys

# With custom hours
python manage.py mark_abandoned_surveys --hours 48

# Dry run (preview without changes)
python manage.py mark_abandoned_surveys --dry-run

# Via Celery (scheduled)
from apps.surveys.tasks import mark_abandoned_surveys
mark_abandoned_surveys.delay(hours=24)
```

**Configuration:**
Add to Django settings:
```python
# Number of hours before marking survey as abandoned
SURVEY_ABANDONMENT_HOURS = 24
```

**Celery Beat Schedule:**
```python
# In config/celery.py
app.conf.beat_schedule = {
    'mark-abandoned-surveys': {
        'task': 'apps.surveys.tasks.mark_abandoned_surveys',
        'schedule': crontab(hour=2, minute=0),  # Run daily at 2 AM
        'kwargs': {'hours': 24}
    }
}
```

**Code Locations:**
- Task: `apps/surveys/tasks.py:mark_abandoned_surveys()`
- Command: `apps/surveys/management/commands/mark_abandoned_surveys.py`

## Tracking Flow

### Survey Send
```
JourneyStage → create_survey_instance()
  ↓
SurveyInstance created with:
  - status = 'sent'
  - sent_at = now()
  - open_count = 0
  - time_spent_seconds = 0
```

### Survey Open
```
Patient opens link → survey_form() view
  ↓
Create SurveyTracking(event_type='page_view')
  ↓
Update SurveyInstance:
  - open_count += 1
  - last_opened_at = now()
  - status = 'viewed'
```

### Survey Progress
```
Patient answers questions
  ↓
Create SurveyTracking(event_type='question_answered')
  ↓
Update SurveyInstance:
  - status = 'in_progress'
  - time_spent_seconds = cumulative time
```

### Survey Completion
```
Patient submits survey → thank_you() view
  ↓
Create SurveyTracking(event_type='survey_completed')
  ↓
Update SurveyInstance:
  - status = 'completed'
  - completed_at = now()
  - total_score calculated
```

### Survey Abandonment
```
Survey sent → not opened after 24h → scheduled task
  ↓
Update SurveyInstance:
  - status = 'abandoned'
  - Create SurveyTracking(event_type='survey_abandoned')
```

## Usage Examples

### Get Engagement Statistics
```python
from apps.surveys.analytics import get_survey_engagement_stats

stats = get_survey_engagement_stats(
    hospital_id=1,
    days=30
)

print(f"Open rate: {stats['open_rate']}%")
print(f"Completion rate: {stats['completion_rate']}%")
print(f"Avg completion time: {stats['avg_completion_time_minutes']} minutes")
```

### Get Patient Timeline
```python
from apps.surveys.analytics import get_patient_survey_timeline

timeline = get_patient_survey_timeline(patient_id=123)

for entry in timeline:
    print(f"Survey: {entry['survey_name']}")
    print(f"Sent: {entry['sent_at']}")
    print(f"Completed: {entry['completed_at']}")
    print(f"Time to complete: {entry['time_to_complete_minutes']} minutes")
```

### Access via API
```bash
# Get engagement stats
curl -H "Authorization: Bearer <token>" \
  "http://localhost:8000/api/surveys/api/analytics/engagement_stats/?hospital_id=1&days=30"

# Get patient timeline
curl -H "Authorization: Bearer <token>" \
  "http://localhost:8000/api/surveys/api/analytics/patient_timeline/?patient_id=123"

# Get abandonment analysis
curl -H "Authorization: Bearer <token>" \
  "http://localhost:8000/api/surveys/api/analytics/abandonment_analysis/?days=30"
```

## Database Schema

### SurveyInstance (New Fields)
```sql
ALTER TABLE surveys_surveyinstance
ADD COLUMN open_count INTEGER DEFAULT 0,
ADD COLUMN last_opened_at TIMESTAMP,
ADD COLUMN time_spent_seconds INTEGER DEFAULT 0,
MODIFY COLUMN status ENUM('sent', 'viewed', 'in_progress', 'completed', 'abandoned', 'expired', 'cancelled');
```

### SurveyTracking (New Table)
```sql
CREATE TABLE surveys_surveytracking (
    id BIGINT PRIMARY KEY AUTO_INCREMENT,
    survey_instance_id BIGINT NOT NULL REFERENCES surveys_surveyinstance(id),
    event_type VARCHAR(20) NOT NULL,
    time_on_page INTEGER,
    total_time_spent INTEGER DEFAULT 0,
    current_question INTEGER,
    user_agent TEXT,
    ip_address VARCHAR(45),
    device_type VARCHAR(50),
    browser VARCHAR(50),
    country VARCHAR(100),
    city VARCHAR(100),
    metadata JSON,
    created_at TIMESTAMP DEFAULT CURRENT_TIMESTAMP,
    INDEX idx_survey_instance_created (survey_instance_id, created_at DESC),
    INDEX idx_event_type_created (event_type, created_at DESC),
    INDEX idx_ip_address (ip_address)
);
```

## Performance Considerations

### Indexing
- Composite index on `survey_instance_id` and `created_at` for fast timeline queries
- Index on `event_type` for event-based filtering
- Index on `ip_address` for location-based analysis

### Caching
- Consider caching engagement stats for frequently accessed periods
- Implement Redis caching for dashboard data

### Data Retention
- SurveyTracking events can be archived after 90 days
- Aggregate daily statistics for long-term reporting

## Future Enhancements

1. **Real-time Analytics**: WebSocket integration for live updates
2. **Geographic Dashboard**: Map visualization of survey responses
3. **Predictive Analytics**: ML model to predict completion likelihood
4. **A/B Testing**: Track engagement with different survey designs
5. **Integration**: Export data to analytics platforms (Google Analytics, Mixpanel)

## Testing

Run the included test suite:
```bash
python manage.py test apps.surveys.tests.test_analytics
```

## Dependencies

Added to `requirements.txt`:
- `user-agents==2.2.0` - Browser/device detection
- `ua-parser==1.0.1` - User agent parsing

## Migration

Run the migration:
```bash
python manage.py migrate surveys
```

## Support

For questions or issues, refer to:
- API documentation: `/api/docs/`
- Admin interface: `/admin/surveys/`
- Code comments in `apps/surveys/analytics.py`