HH/docs/STAFF_HIERARCHY_IMPORT_GUIDE.md

# Staff Hierarchy Data Import and Visualization Guide

## Overview

This guide explains how to import staff data with hierarchy relationships from CSV and visualize it using the PX360 system's two hierarchy visualization options.

## Understanding Your Data

Your CSV data structure:
```
Staff ID,Name,Location,Department,Section,Subsection,AlHammadi Job Title,Country,Gender,Manager
4,ABDULAZIZ SALEH ALHAMMADI,Nuzha,Senior Management Offices,COO Office,,Chief Operating Officer,Saudi Arabia,Male,2 - MOHAMMAD SALEH AL HAMMADI
```

### Key Fields and Mapping

| CSV Column | Staff Model Field | Description |
|------------|------------------|-------------|
| Staff ID | `employee_id` | Unique staff identifier |
| Name | `first_name`, `last_name` | Staff member name (split) |
| Location | Not used | Physical location (not stored) |
| Department | `department` | Department name (lookup/create) |
| Section | Not used | Sub-department (not stored) |
| Subsection | Not used | Further subdivision (not stored) |
| AlHammadi Job Title | `job_title` | Job title |
| Country | Not used | Country (not stored) |
| Gender | Not used | Gender (not stored) |
| Manager | `report_to` | Manager ID and name (ID part used) |

### Manager Field Parsing

The Manager field format: `"ID - NAME"`

**Example:**
```
2 - MOHAMMAD SALEH AL HAMMADI
```

The system extracts:
- **Manager ID:** `2`
- Used to link to the manager's Staff record via `report_to` field

## Staff Model Structure

The PX360 `Staff` model includes:

```python
class Staff(models.Model):
    employee_id = models.CharField(max_length=50, unique=True)
    first_name = models.CharField(max_length=100)
    last_name = models.CharField(max_length=100)
    job_title = models.CharField(max_length=200)
    hospital = models.ForeignKey(Hospital, on_delete=models.CASCADE)
    department = models.ForeignKey(Department, on_delete=models.CASCADE)
    report_to = models.ForeignKey('self', null=True, blank=True, on_delete=models.SET_NULL)
    # ... other fields
```

## Import Methods

### Option 1: Using the Import Command

The system includes a management command for importing CSV data.

**Location:** `apps/organizations/management/commands/import_staff_csv.py`

**Usage:**
```bash
python manage.py import_staff_csv path/to/staff_data.csv
```

**Features:**
- Automatically creates hospitals and departments if needed
- Parses manager relationships
- Creates staff records
- Generates unique IDs for new records
- Validates data integrity

**Example:**
```bash
python manage.py import_staff_csv sample_staff_data.csv
```

### Option 2: Using Django Admin

1. Navigate to Django Admin (`/admin/`)
2. Go to **Organizations → Staff**
3. Click **Import** (if import extension is installed)
4. Upload CSV file
5. Map columns to model fields
6. Import data

### Option 3: Manual Entry via Web Interface

1. Navigate to `/organizations/staff/`
2. Click **Create Staff**
3. Fill in the form:
   - **Employee ID:** From CSV "Staff ID" column
   - **First Name:** First part of "Name"
   - **Last Name:** Second part of "Name"
   - **Job Title:** From "AlHammadi Job Title"
   - **Hospital:** Select appropriate hospital
   - **Department:** Select or create department
   - **Report To:** Select manager (search by ID or name)
4. Save

## Data Import Process

### Step 1: Prepare Your CSV

Ensure your CSV file:
- Has the correct headers
- Uses UTF-8 encoding
- Has consistent formatting
- Contains valid manager IDs

### Step 2: Create Reference Data

Before importing, ensure:
1. **Hospital exists** (or specify in import)
2. **Departments exist** (or let import create them)

Create hospital:
```python
from apps.organizations.models import Hospital
hospital = Hospital.objects.create(
    name="AlHammadi Hospital",
    code="ALH",
    status="active"
)
```

### Step 3: Run Import Command

```bash
python manage.py import_staff_csv sample_staff_data.csv
```

### Step 4: Verify Import

Check imported data:
```python
from apps.organizations.models import Staff

# Count imported staff
total = Staff.objects.count()
print(f"Total staff: {total}")

# Check hierarchy
top_managers = Staff.objects.filter(report_to__isnull=True)
print(f"Top managers: {top_managers.count()}")

# Check a specific staff
staff = Staff.objects.get(employee_id="4")
print(f"{staff.first_name} {staff.last_name}")
print(f"Manager: {staff.report_to}")
print(f"Direct reports: {staff.direct_reports.count()}")
```

## Visualizing Hierarchy

The PX360 system provides **two** hierarchy visualization options:

### Option A: HTML-Based Hierarchy

**URL:** `/organizations/staff/hierarchy/`

**Features:**
- Static tree structure
- Server-side expand/collapse
- Search functionality
- Filters by hospital and department
- Print-friendly

**Access:**
```python
# Direct URL
/organizations/staff/hierarchy/

# From sidebar: Organizations → Staff → Hierarchy
```

### Option B: D3.js Interactive Hierarchy ⭐ NEW

**URL:** `/organizations/staff/hierarchy/d3/`

**Features:**
- Interactive tree visualization
- Client-side expand/collapse (instant)
- Three layout options (horizontal, vertical, radial)
- Zoom and pan support
- Real-time search
- Node sizing by team size or hierarchy level
- Tooltips with detailed information
- Double-click to view staff details
- Smooth animations

**Access:**
```python
# Direct URL
/organizations/staff/hierarchy/d3/

# From sidebar: Organizations → Staff → Hierarchy (D3)
```

## Hierarchy Visualization Comparison

| Feature | HTML-Based | D3.js |
|---------|-----------|-------|
| **Interactivity** | Basic (page reloads) | High (instant) |
| **Expand/Collapse** | Server-side | Client-side |
| **Search** | Server-side (reload) | Client-side (instant) |
| **Zoom/Pan** | ❌ | ✅ |
| **Layouts** | 1 (fixed) | 3 (switchable) |
| **Animations** | ❌ | ✅ |
| **Tooltips** | ❌ | ✅ |
| **Export** | ✅ (print) | Limited |
| **Performance** | Good (small) | Good (all sizes) |
| **Best For** | Printing, reports | Exploration, analysis |

## Using the Hierarchy Views

### HTML-Based Hierarchy

1. **View Full Hierarchy:**
   - Go to `/organizations/staff/hierarchy/`
   - See tree structure with expandable nodes

2. **Search for Staff:**
   - Enter name or ID in search box
   - Click Search
   - Navigate to staff in hierarchy

3. **Filter by Hospital/Department:**
   - Use dropdown filters
   - Click Apply Filters
   - Hierarchy updates

4. **View Staff Details:**
   - Click on staff name
   - Opens staff detail page

### D3.js Interactive Hierarchy

1. **View Full Hierarchy:**
   - Go to `/organizations/staff/hierarchy/d3/`
   - Chart loads with top-level managers
   - Other levels collapsed by default

2. **Expand/Collapse Nodes:**
   - **Single click** on any node to toggle children
   - Use **Expand All** button to show entire organization
   - Use **Collapse All** to show only top managers

3. **Navigate the Chart:**
   - **Mouse wheel:** Zoom in/out
   - **Click & drag:** Pan around the chart
   - **Reset View:** Return to default position

4. **Search Staff:**
   - Type name or ID in search box
   - Press **Enter**
   - Chart auto-navigates to found staff
   - Found node highlighted in red

5. **Change Layout:**
   - Select from dropdown:
     - **Horizontal:** Traditional tree (left to right)
     - **Vertical:** Top-down tree
     - **Radial:** Circular layout
   - Chart reorganizes instantly

6. **Adjust Node Sizes:**
   - Select from dropdown:
     - **Fixed Size:** All nodes same size
     - **Team Size:** Nodes sized by number of direct reports
     - **Level Size:** Nodes sized by hierarchy level
   - Visual updates immediately

7. **View Staff Details:**
   - **Double-click** on any node
   - Opens staff detail page at `/organizations/staff/{id}/`

## Hierarchy Statistics

Both views display:

- **Total Staff:** Number of staff in hierarchy
- **Top Managers:** Number of staff without a manager
- **Average Depth:** Average levels in hierarchy (D3 only)

## API Endpoint

The D3 visualization uses a REST API endpoint:

**URL:** `/organizations/api/staff/hierarchy/`

**Method:** GET

**Parameters:**
- `hospital` (optional): Filter by hospital ID
- `department` (optional): Filter by department ID

**Response:**
```json
{
  "hierarchy": [
    {
      "id": "staff_uuid",
      "name": "Full Name",
      "employee_id": "EMP123",
      "job_title": "Job Title",
      "hospital": "Hospital Name",
      "department": "Department Name",
      "status": "active",
      "staff_type": "type",
      "team_size": 5,
      "children": [...]
    }
  ],
  "statistics": {
    "total_staff": 100,
    "top_managers": 3
  }
}
```

## Troubleshooting

### Import Issues

**Problem:** Manager ID not found

**Solution:**
1. Ensure all manager IDs exist in the Staff ID column
2. Import managers before their direct reports
3. Check ID format (must match exactly)

**Problem:** Department not found

**Solution:**
1. Create departments before importing
2. Or let the import command auto-create departments
3. Check department name spelling

### Hierarchy Issues

**Problem:** Hierarchy shows incorrect relationships

**Solution:**
1. Verify `report_to` field is correctly set
2. Check manager IDs in CSV match staff IDs
3. Re-run import if needed

**Problem:** Staff not appearing in hierarchy

**Solution:**
1. Check staff status is "active"
2. Verify staff belongs to current hospital/department
3. Check RBAC permissions

**Problem:** D3 chart not loading

**Solution:**
1. Check browser console for JavaScript errors
2. Verify API endpoint is accessible
3. Ensure D3.js CDN is reachable
4. Check network tab for failed requests

## Best Practices

### Data Quality

1. **Consistent IDs:** Ensure Staff IDs are unique and consistent
2. **Complete Manager Chains:** All staff should have valid manager IDs (except top-level)
3. **Standardized Names:** Use consistent name formatting
4. **Valid Departments:** Ensure departments exist or can be created

### Hierarchy Structure

1. **Clear Reporting Lines:** Avoid circular references
2. **Reasonable Depth:** Keep hierarchy depth manageable (5-10 levels)
3. **Span of Control:** Consider manager workload (typically 5-15 direct reports)

### Visualization

1. **Use D3 for Exploration:** Best for interactive exploration
2. **Use HTML for Printing:** Best for reports and printing
3. **Filter Large Hierarchies:** Use hospital/department filters for large organizations

## Advanced Usage

### Custom Hierarchy Views

You can create custom hierarchy views by filtering:

```python
# View only medical staff
/organizations/staff/hierarchy/d3/?staff_type=medical

# View specific department
/organizations/staff/hierarchy/d3/?department=dept_id
```

### Programmatic Access

```python
from apps.organizations.models import Staff

# Get hierarchy tree
def get_hierarchy(staff):
    """Get staff hierarchy as nested dict"""
    return {
        'staff': staff,
        'reports': [
            get_hierarchy(report) 
            for report in staff.direct_reports.all()
        ]
    }

# Get top-level managers
top_managers = Staff.objects.filter(report_to__isnull=True)

# Get hierarchy depth
def get_hierarchy_depth(staff):
    """Calculate hierarchy depth from staff"""
    if not staff.report_to:
        return 1
    return 1 + get_hierarchy_depth(staff.report_to)
```

### Export Hierarchy

```bash
# Export staff to CSV
python manage.py dumpdata organizations.Staff > staff_export.json

# Or use Django admin export feature
```

## Documentation References

- **Staff Model:** `apps/organizations/models.py`
- **Import Command:** `apps/organizations/management/commands/import_staff_csv.py`
- **HTML Hierarchy View:** `apps/organizations/ui_views.py` - `staff_hierarchy()`
- **D3 Hierarchy View:** `apps/organizations/ui_views.py` - `staff_hierarchy_d3()`
- **D3 API:** `apps/organizations/views.py` - `StaffViewSet.hierarchy()`
- **D3 Documentation:** `docs/D3_HIERARCHY_INTEGRATION.md`

## Support

For issues or questions:
1. Check browser console for JavaScript errors (D3)
2. Review Django logs for import errors
3. Verify data integrity in Django Admin
4. Contact development team with details

## Summary

The PX360 system provides a complete solution for:
1. **Importing** staff data with hierarchy from CSV
2. **Storing** hierarchical relationships using `report_to` field
3. **Visualizing** hierarchies in two ways:
   - HTML-based (static, print-friendly)
   - D3.js-based (interactive, feature-rich)

Choose the visualization that best fits your use case:
- **HTML:** For reports, printing, simple viewing
- **D3.js:** For exploration, analysis, interactive use