Scale Considerations
When performing large-scale transfers or migrations to SharePoint, several factors affect performance and success. This page covers scale considerations and best practices.
Understanding Scale Challenges
Common Scale Issues
| Issue |
Cause |
Impact |
| Timeouts |
Long-running operations |
Transfer stops |
| Token expiration |
OAuth tokens expire |
Auth failures |
| Network issues |
Connection problems |
Interrupted transfers |
| Rate limiting |
Too many API calls |
Throttled requests |
Factors Affecting Scale
| Factor |
Impact |
| Number of files |
More files = longer time |
| File sizes |
Large files = more bandwidth |
| Network speed |
Affects upload time |
| SharePoint limits |
API rate limits |
Token Management
OAuth Token Lifecycle
Token lifetimes are set by Microsoft Entra ID, not by the connector. Typical Entra ID defaults:
| Token Type |
Typical Duration (Entra ID default) |
| Access token |
~1 hour |
| Refresh token |
~90 days |
Token Refresh Strategy
The connector handles token refresh:
- Access token used for API calls
- When token expires, refresh attempted
- New access token obtained
- Operation continues
VerifyAccess Pattern
For custom development/long operations:
// Pattern for maintaining access
Before each batch:
1. Check token validity
2. Refresh if needed
3. Process batch
4. Repeat
Token Issues
| Symptom |
Cause |
Solution |
| Auth fails mid-transfer |
Token expired |
Retry, smaller batches |
| Refresh fails |
Long inactivity |
Re-authenticate |
| Unauthorized errors |
Invalid token |
Reset connection |
Batch Processing
There is no built-in batching mechanism. You create batches manually by setting request-page filters (for example a record number range), running the report, and then moving the filter to the next range.
Why Work in Batches
| Benefit |
Explanation |
| Manageable chunks |
Easier to track progress |
| Error isolation |
An error only stops the current batch, not the whole migration |
| Verification |
Can verify each batch |
Batch Size Recommendations
| Scenario |
Suggested Batch Size |
| Testing |
10-20 records |
| Normal migration |
100-500 records |
| Large files |
Smaller batches |
| Stable connection |
Larger batches |
Batching Strategy
- Divide records into filter ranges
- Run the report for each range
- Verify the range completed
- Continue to the next range
- Log progress
Timeout Handling
Timeout Scenarios
The connector does not configure a specific HTTP timeout -- the platform defaults apply. Timeouts you may encounter come from the Business Central session or the network, typically during very long report runs.
Preventing Timeouts
| Strategy |
Implementation |
| Smaller batches |
Process fewer records per run (narrower filters) |
| Background processing |
Use Session.StartSession |
Timeout Recovery
There is no built-in resume capability. If a run is interrupted:
- Note roughly where it stopped
- Fix any issues
- Re-run the report with filters narrowed to the remaining records
Files already uploaded are not overwritten on a re-run, so overlapping filters are safe -- they just waste time.
Rate Limiting
SharePoint Limits
SharePoint has API rate limits:
| Limit Type |
Description |
| Per-minute |
Requests per minute cap |
| Daily |
Total daily requests |
| Per-resource |
Per-file limits |
Handling Throttling
The connector handles throttling automatically. When SharePoint returns HTTP 429 or 503, the connector reads the Retry-After header, shows a countdown dialog (when a user interface is available), sleeps for the requested time, and then retries the request.
You do not need to react to throttling manually, but you can reduce how often it occurs:
- Run large operations during off-peak hours
- Process smaller filtered sets at a time
Avoiding Throttling
| Practice |
Benefit |
| Space out large runs |
Avoid bursts |
| Off-peak hours |
Less contention |
| Monitor usage |
Catch issues early |
Network Considerations
| Factor |
Optimization |
| Bandwidth |
Use during low-usage periods |
| Latency |
Batch operations |
| Stability |
Wired vs wireless |
Processing Tips
| Tip |
Benefit |
| Background mode |
Doesn't block users |
| Off-hours |
Less system load |
| Dedicated session |
Focused resources |
Monitoring Progress
What to Monitor
| Metric |
Why |
| Records processed |
Track completion |
| Errors |
Identify issues |
| Time elapsed |
Estimate remaining |
| Resources |
System health |
Logging
Keep records of:
- Start time
- Records processed
- Errors encountered
- Completion status
Example Log
| Time |
Action |
Status |
| 10:00 |
Start batch 1 (records 1-100) |
Started |
| 10:15 |
Complete batch 1 |
Success |
| 10:16 |
Start batch 2 (records 101-200) |
Started |
| 10:25 |
Error on record 150 |
Failed |
| 10:26 |
Re-run batch 2 (filter narrowed to 150-200) |
Restarted |
Large Migration Strategy
Phase 1: Planning
| Task |
Details |
| Inventory |
Count records, files, sizes |
| Timeline |
Estimate duration |
| Resources |
Allocate time, people |
| Testing |
Plan test migrations |
Phase 2: Testing
| Task |
Details |
| Small batch |
Test with 10-20 records |
| Verify |
Confirm results |
| Measure |
Note timing |
| Adjust |
Refine approach |
Phase 3: Execution
| Task |
Details |
| Execute |
Run batches |
| Monitor |
Watch progress |
| Document |
Log everything |
| Verify |
Check each batch |
Phase 4: Completion
| Task |
Details |
| Final verify |
Complete audit |
| Cleanup |
Remove duplicates if any |
| Communicate |
Inform stakeholders |
| Document |
Final documentation |
Recovery Procedures
If Transfer Fails
- Stop - Don't make it worse
- Assess - Understand what failed
- Fix - Address root cause
- Re-run - Run again with filters narrowed to the remaining records
Partial Failure Recovery
| Status |
Action |
| Some succeeded |
Note which ones |
| Identify failed |
List failures |
| Retry failed |
Only those |
| Verify all |
Complete check |
Best Practices Summary
Do
| Practice |
Why |
| Plan ahead |
Avoid surprises |
| Test first |
Catch issues early |
| Use batches |
Manageable chunks |
| Monitor closely |
React quickly |
| Document everything |
Reference later |
Don't
| Avoid |
Why |
| Rush |
Quality issues |
| Ignore errors |
Incomplete migration |
| Skip verification |
Missing files |
| Forget logging |
Can't trace problems |
