HH/docs/SIMULATOR_LOGGING_IMPLEMENTATION.md

Simulator Data Logging Implementation

Overview

This implementation adds persistent data logging and a web-based viewer for the Survey and Patient Journey Simulator. All simulator requests (Email, SMS, and HIS Events) are now automatically logged to the database and can be viewed through a user-friendly interface.

Features

1. Automatic Data Logging

All simulator requests are automatically logged to the database with the following information:

• Email Simulator: Recipient, subject, message preview, status, response data
• SMS Simulator: Recipient phone number, message preview, status, response data
• HIS Journey Events: Patient MRN, encounter ID, hospital code, event type, visit type, department

2. Processing Metrics

Each log entry captures:

• Request timestamp
• Processing time (in milliseconds)
• Request status (success, sent, failed, partial)
• Error messages (if applicable)
• IP address and user agent

3. Web-Based Log Viewer

Access the simulator logs at: /simulator/logs/

Features:

• Statistics Dashboard: View total requests, success rate, average processing time
• Channel Breakdown: See counts for Email, SMS, and HIS Events
• Status Breakdown: Success, failed, and partial request counts
• Advanced Filtering: Filter by channel, status, hospital, visit type, date range
• Search: Search by request ID, patient MRN, recipient, or subject
• Pagination: Handle large datasets with customizable page sizes
• Detailed View: Click any log entry to see full request/response data

4. Log Detail View

Each log entry can be viewed in detail showing:

• Full request payload (formatted JSON)
• Full response data (formatted JSON)
• Related objects (patient, journey, survey if available)
• Error details (if applicable)
• Message preview
• Copy to clipboard functionality

Database Schema

SimulatorRequestLog Model

class SimulatorRequestLog(models.Model):
    # Request identification
    request_id = models.AutoField(primary_key=True)
    timestamp = models.DateTimeField(auto_now_add=True)
    
    # Channel identification
    channel = models.CharField(choices=['email', 'sms', 'his_event'])
    
    # Request data
    payload = models.JSONField(default=dict)
    
    # Response data
    status = models.CharField()  # success, sent, failed, partial
    response_data = models.JSONField(null=True, blank=True)
    
    # Processing details
    processing_time_ms = models.IntegerField(null=True, blank=True)
    error_message = models.TextField(null=True, blank=True)
    
    # Related objects
    patient_id = models.CharField(max_length=100, null=True, blank=True)
    journey_id = models.CharField(max_length=100, null=True, blank=True)
    survey_id = models.CharField(max_length=100, null=True, blank=True)
    hospital_code = models.CharField(max_length=50, null=True, blank=True)
    
    # Email/SMS specific
    recipient = models.CharField(max_length=255, null=True, blank=True)
    subject = models.CharField(max_length=500, null=True, blank=True)
    message_preview = models.TextField(null=True, blank=True)
    
    # HIS event specific
    event_type = models.CharField(max_length=100, null=True, blank=True)
    visit_type = models.CharField(max_length=50, null=True, blank=True)
    department = models.CharField(max_length=100, null=True, blank=True)
    
    # Metadata
    ip_address = models.GenericIPAddressField(null=True, blank=True)
    user_agent = models.TextField(null=True, blank=True)

API Endpoints

Logging Endpoints (Automatic)

These endpoints automatically log all requests:

• POST /simulator/api/email/ - Email simulator
• POST /simulator/api/sms/ - SMS simulator
• POST /simulator/api/his-events/ - HIS events handler

UI Endpoints

• GET /simulator/logs/ - View all simulator logs
• GET /simulator/logs/<id>/ - View specific log details
• POST /simulator/logs/clear/ - Clear all logs (admin only)

Utility Endpoints

• GET /simulator/health/ - Simulator health check and statistics
• GET /simulator/reset/ - Reset simulator statistics (in-memory only)

Usage Examples

1. Send Email and View Logs

curl -X POST http://localhost:8000/simulator/api/email/ \
  -H "Content-Type: application/json" \
  -d '{
    "to": "patient@example.com",
    "subject": "Survey Invitation",
    "message": "Please complete our survey..."
  }'

Then view the log at: http://localhost:8000/simulator/logs/

2. Send SMS and View Logs

curl -X POST http://localhost:8000/simulator/api/sms/ \
  -H "Content-Type: application/json" \
  -d '{
    "to": "+966501234567",
    "message": "Your survey is ready..."
  }'

3. Send HIS Events and View Logs

curl -X POST http://localhost:8000/simulator/api/his-events/ \
  -H "Content-Type: application/json" \
  -d '{
    "events": [{
      "encounter_id": "ENC-2024-001",
      "mrn": "MRN-12345",
      "national_id": "1234567890",
      "first_name": "Ahmed",
      "last_name": "Mohammed",
      "phone": "+966501234567",
      "email": "patient@example.com",
      "event_type": "OPD_STAGE_1_REGISTRATION",
      "timestamp": "2024-01-20T10:30:00Z",
      "visit_type": "opd",
      "department": "Cardiology",
      "hospital_code": "ALH-main"
    }]
  }'

4. View Simulator Health

curl http://localhost:8000/simulator/health/

Response:

{
  "status": "healthy",
  "timestamp": "2024-01-20T10:30:00Z",
  "statistics": {
    "total_requests": 15,
    "email_requests": 5,
    "sms_requests": 7,
    "recent_requests": [...]
  }
}

Web Interface

Log List View

Access: /simulator/logs/

Features:

• Statistics cards showing:
  • Total requests
  • Success/failed/partial counts
  • Success rate percentage
  • Average processing time
  • Survey invitations sent
• Channel breakdown (Email, SMS, HIS Events)
• Status breakdown
• Hospital breakdown (for HIS events)
• Advanced filters
• Paginated results table

Log Detail View

Access: /simulator/logs/<id>/

Features:

• Basic information (channel, status, timestamp, processing time)
• Related objects (patient, journey, survey if available)
• Request payload (formatted JSON)
• Response data (formatted JSON)
• Error details (if applicable)
• Message preview
• Copy to clipboard for JSON data

Maintenance

Clearing Logs

To clear all simulator logs (admin only):

# Via UI
Visit /simulator/logs/ and click "Clear All Logs"

# Via API
POST /simulator/logs/clear/

Database Management

# Create migrations
uv run python manage.py makemigrations simulator

# Apply migrations
uv run python manage.py migrate simulator

# Reset database (WARNING: Deletes all data)
uv run python manage.py migrate simulator zero
uv run python manage.py migrate simulator

Performance Considerations

1. Indexing: The model includes indexes on frequently queried fields:

   • timestamp
   • channel
   • status
   • patient_id
   • journey_id
   • survey_id
   • hospital_code
   • event_type
   • visit_type

2. Pagination: Use pagination to handle large datasets efficiently

3. Query Optimization: The views use select_related for related objects to avoid N+1 queries

4. JSON Storage: Full request/response data is stored as JSON for flexibility

Security

1. Authentication: Log viewer requires user login (@login_required decorator)

2. Authorization: Clear logs feature restricted to superusers and PX admins

3. IP Tracking: Client IP addresses are logged for auditing

4. User Agent: User agent information is captured for debugging

Testing

Test Email Simulator Logging

curl -X POST http://localhost:8000/simulator/api/email/ \
  -H "Content-Type: application/json" \
  -d '{
    "to": "test@example.com",
    "subject": "Test Email",
    "message": "This is a test email"
  }'

Verify log at: http://localhost:8000/simulator/logs/

Test SMS Simulator Logging

curl -X POST http://localhost:8000/simulator/api/sms/ \
  -H "Content-Type: application/json" \
  -d '{
    "to": "+966501234567",
    "message": "This is a test SMS"
  }'

Test HIS Events Logging

curl -X POST http://localhost:8000/simulator/api/his-events/ \
  -H "Content-Type: application/json" \
  -d '{
    "events": [{
      "encounter_id": "TEST-001",
      "mrn": "TEST-MRN",
      "national_id": "1234567890",
      "first_name": "Test",
      "last_name": "Patient",
      "phone": "+966501234567",
      "email": "test@example.com",
      "event_type": "OPD_STAGE_1_REGISTRATION",
      "timestamp": "2024-01-20T10:30:00Z",
      "visit_type": "opd",
      "department": "Cardiology",
      "hospital_code": "ALH-main"
    }]
  }'

Troubleshooting

Logs Not Appearing

1. Check if the database migration was applied:

   uv run python manage.py showmigrations simulator
   

2. Check for errors in logs:

   tail -f logs/django.log
   

3. Verify the model is imported in views.py

Slow Query Performance

1. Add more indexes if needed
2. Use date range filters to reduce dataset size
3. Consider archiving old logs

Memory Issues

1. Reduce page size in the UI
2. Implement log rotation/archival
3. Add cleanup jobs to remove old logs

Future Enhancements

1. Export Functionality: Export logs to CSV/Excel
2. Log Archival: Automatic archival of old logs
3. Analytics Dashboard: Charts and graphs for log statistics
4. Real-time Updates: WebSocket support for real-time log updates
5. Alerting: Email/SMS alerts for failed requests
6. Performance Metrics: More detailed performance tracking
7. Log Search: Full-text search across all log fields
8. API Documentation: Swagger/OpenAPI documentation for endpoints

Files Modified/Created

Created:

• apps/simulator/models.py - Database model
• apps/simulator/ui_views.py - Web views
• templates/simulator/log_list.html - List view template
• templates/simulator/log_detail.html - Detail view template

Modified:

• apps/simulator/views.py - Added logging to all endpoints
• apps/simulator/urls.py - Added UI routes

Summary

This implementation provides a complete solution for logging and viewing simulator data. All requests are automatically captured, stored in the database, and can be viewed through a user-friendly web interface with advanced filtering and search capabilities. The system is designed to be performant, secure, and maintainable.