HH/docs/STAFF_HIERARCHY_INTEGRATION_SUMMARY.md

# Staff Hierarchy Integration - Complete Summary

## Executive Summary

The PX360 system now includes comprehensive support for importing, storing, and visualizing staff hierarchy data from CSV files. This document provides a complete overview of the integration process and available features.

## Your Data Structure

### CSV Format
```
Staff ID,Name,Location,Department,Section,Subsection,AlHammadi Job Title,Country,Gender,Manager
4,ABDULAZIZ SALEH ALHAMMADI,Nuzha,Senior Management Offices,COO Office,,Chief Operating Officer,Saudi Arabia,Male,2 - MOHAMMAD SALEH AL HAMMADI
1049,VARGHESE   NINAN,Nuzha,Human Resource ,Employee Relations,,Porter,India,Male,1053 - MAJID SALEM SAEED ALNAHDI
1053,MAJID SALEM SAEED ALNAHDI,Nuzha,Human Resource ,Administration,,Human Resources Manager ,Saudi Arabia,Male,2 - MOHAMMAD SALEH AL HAMMADI
```

### Data Mapping

| CSV Field | PX360 Field | Notes |
|-----------|-------------|-------|
| Staff ID | `employee_id` | Unique identifier (required) |
| Name | `first_name`, `last_name` | Split into first/last name |
| Location | ❌ Not stored | Physical location |
| Department | `department` | Department name (lookup/create) |
| Section | ❌ Not stored | Sub-department |
| Subsection | ❌ Not stored | Further subdivision |
| AlHammadi Job Title | `job_title` | Job title |
| Country | ❌ Not stored | Country |
| Gender | ❌ Not stored | Gender |
| Manager | `report_to` | Manager ID (extracted from "ID - NAME") |

### Manager Field Parsing

**Format:** `"ID - NAME"`

**Example:** `2 - MOHAMMAD SALEH AL HAMMADI`

**System Behavior:**
1. Extracts ID: `2`
2. Looks up Staff record with `employee_id = "2"`
3. Sets `report_to` field to that Staff record
4. Creates hierarchical relationship

## Integration Architecture

### 1. Data Storage

**Model:** `apps.organizations.models.Staff`

**Key Hierarchy Fields:**
```python
employee_id = CharField(unique=True)           # From CSV "Staff ID"
first_name = CharField()                         # From CSV "Name" (first part)
last_name = CharField()                          # From CSV "Name" (last part)
job_title = CharField()                          # From CSV "AlHammadi Job Title"
department = ForeignKey(Department)              # From CSV "Department"
report_to = ForeignKey('self')                   # From CSV "Manager" (ID part)
```

**Relationship:** `report_to` creates self-referential foreign key for hierarchy

### 2. Data Import

**Command:** `python manage.py import_staff_csv path/to/file.csv`

**Location:** `apps/organizations/management/commands/import_staff_csv.py`

**Process:**
1. Read CSV file
2. Parse each row
3. Extract manager ID from Manager field
4. Create/update Staff records
5. Set `report_to` relationships
6. Validate data integrity

### 3. Hierarchy Visualization Options

The system provides **two** visualization approaches:

#### Option A: HTML-Based Hierarchy (Original)

**URL:** `/organizations/staff/hierarchy/`

**Characteristics:**
- Server-side rendering
- Static tree structure
- Page reloads for interactions
- Print-friendly
- Good for reports

**Technology:** Django templates + Bootstrap

**Documentation:** `docs/STAFF_HIERARCHY_VIEW.md`

#### Option B: D3.js Interactive Hierarchy ⭐ NEW

**URL:** `/organizations/staff/hierarchy/d3/`

**Characteristics:**
- Client-side rendering
- Interactive tree visualization
- Instant interactions (no reloads)
- Multiple layouts
- Zoom & pan
- Rich interactivity

**Technology:** D3.js v7.9.0 + JavaScript

**Documentation:** `docs/D3_HIERARCHY_INTEGRATION.md`

## Complete Workflow

### Step 1: Prepare Data

1. Ensure CSV has correct headers
2. Verify all Manager IDs exist in Staff ID column
3. Check department names are consistent
4. Save as UTF-8 encoded CSV

### Step 2: Import Data

```bash
# Option 1: Use import command
python manage.py import_staff_csv sample_staff_data.csv

# Option 2: Use Django Admin
# Navigate to /admin/organizations/staff/
# Use import feature

# Option 3: Manual entry
# Navigate to /organizations/staff/create/
# Enter data manually
```

### Step 3: Verify Data

```python
# Check import
from apps.organizations.models import Staff

# Count staff
print(f"Total staff: {Staff.objects.count()}")

# Check top managers
print(f"Top managers: {Staff.objects.filter(report_to__isnull=True).count()}")

# Check specific staff
staff = Staff.objects.get(employee_id="4")
print(f"{staff.first_name} reports to {staff.report_to}")
print(f"Direct reports: {staff.direct_reports.count()}")
```

### Step 4: Visualize Hierarchy

#### HTML-Based View
```
URL: /organizations/staff/hierarchy/
- See tree structure
- Click to expand/collapse (page reloads)
- Search by name or ID
- Filter by hospital/department
```

#### D3.js Interactive View
```
URL: /organizations/staff/hierarchy/d3/
- Interactive tree visualization
- Click to expand/collapse (instant)
- Three layouts: horizontal, vertical, radial
- Zoom and pan support
- Real-time search
- Node sizing options
- Tooltips with details
- Double-click to view staff details
```

## Feature Comparison

| Feature | HTML | D3.js |
|---------|------|-------|
| **Rendering** | Server-side | Client-side |
| **Interactivity** | Basic (reload) | High (instant) |
| **Layouts** | 1 fixed | 3 switchable |
| **Zoom/Pan** | ❌ | ✅ |
| **Animations** | ❌ | ✅ |
| **Tooltips** | ❌ | ✅ |
| **Search** | Server-side | Client-side |
| **Export** | ✅ Print | Limited |
| **Best For** | Reports | Exploration |

## Technical Implementation

### 1. Import Command

**File:** `apps/organizations/management/commands/import_staff_csv.py`

**Key Functions:**
- `handle()`: Main command entry point
- `parse_manager()`: Extract ID from "ID - NAME" format
- `create_or_update_staff()`: Create/update staff records
- `create_department()`: Auto-create departments
- `set_manager_relationship()`: Set `report_to` field

### 2. Hierarchy API

**File:** `apps/organizations/views.py`

**Endpoint:** `/organizations/api/staff/hierarchy/`

**Method:** GET

**Parameters:**
- `hospital` (optional): Filter by hospital
- `department` (optional): Filter by department

**Response:** JSON hierarchy tree with statistics

### 3. D3 Visualization

**File:** `templates/organizations/staff_hierarchy_d3.html`

**Key Components:**
- `fetchHierarchyData()`: Fetch data from API
- `update()`: Render/update visualization
- `diagonal()`: Create curved links
- `click()`: Toggle node expansion
- `getNodeRadius()`: Calculate node sizes

**Features:**
- Three tree layouts (horizontal, vertical, radial)
- Zoom and pan behavior
- Search with auto-navigation
- Expand/collapse animations
- Responsive sizing

## Documentation Structure

### Integration Guides

1. **STAFF_HIERARCHY_IMPORT_GUIDE.md**
   - How to import CSV data
   - Data mapping details
   - Troubleshooting import issues
   - Usage examples

2. **D3_HIERARCHY_INTEGRATION.md**
   - D3.js implementation details
   - Features and capabilities
   - User guide
   - Customization options
   - Troubleshooting

3. **STAFF_HIERARCHY_VIEW.md** (existing)
   - HTML-based hierarchy view
   - Original implementation

### Code Documentation

- **apps/organizations/models.py** - Staff model definition
- **apps/organizations/views.py** - API endpoint
- **apps/organizations/ui_views.py** - View functions
- **apps/organizations/urls.py** - URL routing
- **templates/organizations/staff_hierarchy.html** - HTML view
- **templates/organizations/staff_hierarchy_d3.html** - D3 view
- **apps/organizations/management/commands/import_staff_csv.py** - Import command

## Quick Start

### Import Your Data

```bash
# 1. Prepare CSV file
# Ensure format matches your example

# 2. Import data
python manage.py import_staff_csv your_staff_data.csv

# 3. Verify import
python manage.py shell
>>> from apps.organizations.models import Staff
>>> Staff.objects.count()
>>> Staff.objects.filter(report_to__isnull=True).count()
```

### View Hierarchy

#### HTML-Based View
```
1. Navigate to: /organizations/staff/hierarchy/
2. See tree structure with top-level managers
3. Click to expand/collapse nodes
4. Use search to find specific staff
5. Filter by hospital/department
```

#### D3 Interactive View
```
1. Navigate to: /organizations/staff/hierarchy/d3/
2. See interactive chart
3. Click nodes to expand/collapse
4. Use mouse wheel to zoom
5. Drag to pan
6. Search for staff (auto-navigate)
7. Change layout from dropdown
8. Adjust node sizes from dropdown
9. Double-click nodes for details
```

## Example Data Flow

### CSV Import

```
CSV Row:
  Staff ID: 1053
  Name: MAJID SALEM SAEED ALNAHDI
  Manager: 2 - MOHAMMAD SALEH AL HAMMADI
  Job Title: Human Resources Manager
  Department: Human Resource

↓ Import Command

Staff Record Created:
  employee_id: "1053"
  first_name: "MAJID SALEM SAEED"
  last_name: "ALNAHDI"
  job_title: "Human Resources Manager"
  report_to: Staff.objects.get(employee_id="2")

↓ Hierarchy Built

Tree Structure:
  Mohammad Saleh Al Hammadi (ID: 2)
  └── Majid Salem Saeed Alnahdi (ID: 1053)
      └── Varghese Ninan (ID: 1049)
```

### D3 Visualization

```
API Request: GET /organizations/api/staff/hierarchy/

↓

Query Database:
  - Get all staff (filtered by permissions)
  - Build hierarchy tree
  - Calculate team sizes
  - Compute statistics

↓

Return JSON:
  {
    "hierarchy": [...],
    "statistics": {...}
  }

↓

D3.js Renders:
  - Interactive tree chart
  - With zoom, pan, search
  - Multiple layout options
  - Smooth animations
```

## Common Use Cases

### 1. View Entire Organization

**D3 View:** `/organizations/staff/hierarchy/d3/`
- Click "Expand All"
- See complete organization structure
- Use zoom/pan to navigate
- Good for understanding full hierarchy

### 2. Find Specific Staff Member

**D3 View:** `/organizations/staff/hierarchy/d3/`
- Type name or ID in search box
- Press Enter
- Chart auto-navigates to staff
- Node highlighted in red
- Instant feedback

### 3. Print Hierarchy Report

**HTML View:** `/organizations/staff/hierarchy/`
- Expand desired sections
- Use browser print (Ctrl/Cmd + P)
- Static layout prints well
- Good for reports

### 4. Analyze Team Structure

**D3 View:** `/organizations/staff/hierarchy/d3/`
- Set node size to "Team Size"
- Larger nodes = more direct reports
- Identify managers with large teams
- Spot potential bottlenecks

### 5. Compare Departments

**D3 View:** `/organizations/staff/hierarchy/d3/`
- Use department filter (if available)
- Switch between departments
- Compare hierarchy structures
- Identify organizational patterns

## Troubleshooting

### Import Issues

**Problem:** "Manager ID not found"
- **Solution:** Ensure manager IDs exist in Staff ID column
- **Check:** Import managers before their reports

**Problem:** "Department not found"
- **Solution:** Create departments first, or let import auto-create
- **Check:** Department name spelling

### Hierarchy Issues

**Problem:** Wrong relationships
- **Solution:** Verify `report_to` field in database
- **Check:** Manager IDs in CSV match Staff IDs

**Problem:** Staff missing from hierarchy
- **Solution:** Check staff status is "active"
- **Check:** Verify hospital/department permissions
- **Check:** RBAC settings

### D3 View Issues

**Problem:** Chart not loading
- **Solution:** Check browser console for errors
- **Check:** API endpoint is accessible
- **Check:** D3.js CDN is reachable

**Problem:** Search not working
- **Solution:** Verify staff exist in hierarchy
- **Check:** JavaScript console for errors
- **Try:** Full name instead of partial

## Best Practices

### Data Quality
1. ✅ Use consistent Staff IDs
2. ✅ Complete manager chains
3. ✅ Standardized department names
4. ✅ Valid job titles

### Hierarchy Structure
1. ✅ Clear reporting lines
2. ✅ Avoid circular references
3. ✅ Reasonable depth (5-10 levels)
4. ✅ Appropriate span of control (5-15 direct reports)

### Visualization
1. ✅ Use D3 for exploration and analysis
2. ✅ Use HTML for printing and reports
3. ✅ Filter large hierarchies
4. ✅ Choose appropriate layout for your needs

## API Reference

### Hierarchy API

**Endpoint:** `/organizations/api/staff/hierarchy/`

**Method:** GET

**Parameters:**
- `hospital` (optional): Filter by hospital ID
- `department` (optional): Filter by department ID

**Response:**
```json
{
  "hierarchy": [
    {
      "id": "uuid",
      "name": "Full Name",
      "employee_id": "EMP123",
      "job_title": "Job Title",
      "hospital": "Hospital Name",
      "department": "Department Name",
      "status": "active",
      "staff_type": "type",
      "team_size": 5,
      "children": [...]
    }
  ],
  "statistics": {
    "total_staff": 100,
    "top_managers": 3
  }
}
```

## Future Enhancements

### Potential Additions

1. **Export Features**
   - Export D3 chart as PNG/SVG
   - Export as PDF
   - Download hierarchy data as JSON

2. **Advanced Filtering**
   - Filter by staff type
   - Filter by status
   - Multi-criteria filters

3. **Additional Visualizations**
   - Sunburst chart
   - Treemap
   - Force-directed graph

4. **Analytics Dashboard**
   - Hierarchy depth analysis
   - Span of control metrics
   - Bottleneck identification
   - Organizational health metrics

5. **Collaboration Features**
   - Shareable hierarchy links
   - Embed in reports
   - Compare hierarchies over time

## Support and Resources

### Documentation
- **Import Guide:** `docs/STAFF_HIERARCHY_IMPORT_GUIDE.md`
- **D3 Integration:** `docs/D3_HIERARCHY_INTEGRATION.md`
- **HTML View:** `docs/STAFF_HIERARCHY_VIEW.md`

### Code Files
- **Import Command:** `apps/organizations/management/commands/import_staff_csv.py`
- **API:** `apps/organizations/views.py` - `StaffViewSet.hierarchy()`
- **Views:** `apps/organizations/ui_views.py`
- **Templates:** `templates/organizations/staff_hierarchy*.html`

### External Resources
- **D3.js:** https://d3js.org/
- **D3 Tree:** https://github.com/d3/d3-hierarchy/tree

## Summary

The PX360 system now provides a complete solution for:

✅ **Importing** staff data with hierarchy from CSV  
✅ **Storing** hierarchical relationships using `report_to` field  
✅ **Visualizing** hierarchies in two powerful ways:
   - HTML-based (static, print-friendly)
   - D3.js-based (interactive, feature-rich)

**Choose the right visualization for your needs:**
- **HTML View:** For reports, printing, simple viewing
- **D3 View:** For exploration, analysis, interactive use

**Access your hierarchy visualizations:**
- HTML: `/organizations/staff/hierarchy/`
- D3: `/organizations/staff/hierarchy/d3/`

**Import your data:**
```bash
python manage.py import_staff_csv your_staff_data.csv
```

The system is ready to handle your staff hierarchy data with powerful visualization capabilities!