System Administrator Interview Q&A Matrix

Start with repadmin /replsummary — this gives a quick overview of replication failures and failure counts across all DCs. If you see errors, open Event Viewer on both DCs and check the Directory Service log and DFS Replication log for specific error codes. Next, confirm time synchronization with w32tm /query /status because a time skew of more than 5 minutes breaks Kerberos and kills replication. Finally, verify network connectivity between the DCs and confirm the NTDS service is running on both.

First, pull the Security event log on the PDC Emulator DC and look for Event ID 4740 (account lockout). This event shows the username and — critically — the caller computer name, which tells you what device is sending the bad password. Common sources are: old saved credentials on a phone or tablet, a Windows service running under a user account with a stale password, or a mapped drive with cached credentials. Once you've identified the source machine, clear or update the credentials. Also confirm AD replication is healthy and time is in sync, because skewed clocks can cause lockouts that look random.

The first thing to check is always DNS — this error almost always means the PC can't resolve the domain name. Open a command prompt on the PC and run nslookup company.local. If it fails, the PC's DNS is pointing to the wrong server (likely a public DNS like 8.8.8.8 instead of an internal domain controller). Set the DNS to a DC's IP and retry. If DNS resolves but join still fails: (1) Confirm the PC can reach the DC on the network — ping dc01.company.local. (2) Check whether a stale computer account with the same name already exists in AD from a previous join — if so, delete it and retry. (3) Check the time skew — Kerberos authentication fails if the PC clock is more than 5 minutes off from the DC. (4) Confirm the account performing the join has permission to add computers (regular domain users can join up to 10 by default, but this may be restricted by GPO).

The core risk is lingering objects — objects that were deleted on other DCs (and passed tombstone lifetime) still exist on the DC that was offline. When it comes back, those "ghost" objects can replicate back and undo legitimate deletions. The fix is: identify lingering objects with repadmin /showrepl errors, then remove them using repadmin /removelingeringobjects on each affected DC. Validate the replication topology with the KCC (Knowledge Consistency Checker). If the DC's state is too far out of sync, the cleanest option is to demote it, clean up its metadata using ntdsutil, and then re-promote it fresh so it replicates a clean copy.

Run dfsrdiag ReplicationState on both DCs to check the current replication state. Check the DFS Replication event log for errors — look specifically for backlog counts and any files in the Conflict and Deleted folder (which means DFSR detected conflicts and quarantined files). Verify that the DFSR service is running, that permissions on SYSVOL are correct, and that the network path between sites is open. If DFSR is in an error state, you may need to perform a non-authoritative restore on the branch DC (set DFSR to pull a fresh copy from HQ) or in serious cases, an authoritative restore if HQ also has corrupt data.

As long as DC1 is still online, you do a graceful transfer — not a seize. You can transfer roles through Active Directory Users and Computers (for RID Master, PDC Emulator, Infrastructure Master), Active Directory Domains and Trusts (for Domain Naming Master), or via PowerShell with Move-ADDirectoryServerOperationMasterRole. Before the move, ensure DC2 is fully synced, healthy, and that the time service is working. After the transfer, document which DC holds which roles, and validate by running netdom query fsmo. A seize (done via ntdsutil) should only be used as a last resort when the original role holder is permanently dead — seizing on a DC that can still come back online will cause a conflict and can break the domain.

Start by checking the Azure AD Connect synchronization service on the sync server — open Synchronization Service Manager and look at the Operations tab for recent export errors. Look for error types: AttributeValueMustBeUnique (duplicate email/UPN/proxyAddress across accounts) or InvalidSoftMatch. Check that the new user accounts are in an OU that is within the sync scope — if the OU they were created in wasn't included in the AAD Connect OU filter, those accounts will never sync. Also verify the AAD Connect service account hasn't expired or lost permissions. To force a sync cycle now: run Start-ADSyncSyncCycle -PolicyType Delta in PowerShell on the sync server, then check for errors again. For attribute conflicts, resolve the duplicate value in on-prem AD (e.g. ensure each user has a unique UPN suffix like @company.com) and re-sync.

Define one AD Site per physical location and create Site Link objects with appropriate cost values and replication schedules — branches with limited bandwidth should replicate outside business hours. Within each site, authentication should stay local so that a WAN outage does not lock users out. For low-bandwidth or high-security branch offices, deploy Read-Only Domain Controllers (RODCs) — they hold a filtered copy of AD, cannot be written to, and reduce the risk if a branch DC is compromised. Place bridgehead server selection under explicit control rather than leaving it to the KCC in large or complex topologies. Enforce a tiered admin model (Tier 0 = domain controllers, Tier 1 = servers, Tier 2 = workstations) to limit lateral movement if any tier is compromised.

This requires an authoritative restore. Steps: (1) Isolate one DC from the network immediately to prevent further replication. (2) Reboot that DC into Directory Services Restore Mode (DSRM). (3) Restore from the most recent System State backup using Windows Server Backup. (4) Before bringing the DC back online, use ntdsutil to mark the deleted OU and its objects as authoritative — this bumps their USN version number so other DCs treat the restored objects as the latest version and replicate them back. (5) Reconnect the DC and verify replication restores the objects across all DCs. (6) Validate that user accounts, group memberships, and passwords are intact.

Install the DNS role on a second domain controller so there are at least two DNS servers for the domain. Configure all client machines and DHCP scopes to point to both DNS servers (primary and secondary) so if one is down, clients automatically fall back. Make sure forwarders are configured on the secondary DNS as well. After the change, test name resolution from a client machine using nslookup or Resolve-DnsName to confirm both servers answer correctly.

Create a DHCP reservation — this ties a specific IP address to the printer's MAC address so it always gets the same IP. The reserved IP should be outside the DHCP exclusion range (or the regular dynamic range) to prevent conflicts. When creating the reservation, also confirm the correct scope options are set — default gateway and DNS server — so the printer can actually communicate on the network. Test by releasing and renewing the printer's DHCP lease to confirm it gets the right IP.

Check the DHCP scope utilization for that branch's subnet — open DHCP Manager, find the scope, and look at the Address Pool vs Address Leases. If utilization is at 100%, the scope is exhausted and no new leases can be issued. Immediate fix: if there's room in the subnet, extend the scope's IP range. If you can't expand, reduce the lease duration so idle devices release addresses faster (right-click scope → Properties → Lease Duration). Also check for a rogue DHCP server on the network — a misconfigured router, a personal hotspot, or even a VM with DHCP enabled can exhaust the pool. Look for duplicate IPs in the DHCP logs (Event ID 1014). Longer term, review whether the /24 subnet is appropriately sized for the number of devices at that site.

Use Conditional Forwarders — a DNS rule that says "for queries about partner.com, forward to these specific DNS servers." This keeps everything else going to your normal resolvers. An alternative is a Stub Zone, which automatically keeps a list of the partner domain's authoritative name servers up to date. On the dynamic updates side, make sure DNS dynamic update is set to Secure only (not unsecured) so only domain-joined machines can register records. Configure DNS scavenging with appropriate aging settings to automatically clean up stale records from machines that left the domain or changed IPs.

Split DNS means you have the same domain name (e.g. autodiscover.company.com) resolving to different IPs internally vs externally. Internally, autodiscover.company.com should point to wherever the user's mailbox is — either the on-prem Exchange server for on-prem mailboxes, or a CNAME to autodiscover.outlook.com for Exchange Online mailboxes. Externally, it should point to Exchange Online via CNAME. For the hybrid scenario, internal DNS should direct migrated users to autodiscover.outlook.com while still routing on-prem mailboxes to the on-prem server. Also confirm the SSL certificate has the correct SAN (Subject Alternative Name) entries covering both namespaces.

The receiving mail servers are doing a reverse DNS (PTR) lookup — they take your mail server's public IP address and look up what hostname it resolves to. If there's no PTR record or it doesn't match your mail server's FQDN (as sent in the EHLO greeting), many servers reject the email as a spam prevention measure. The fix: contact whoever owns the IP block — usually your ISP or datacenter — and ask them to create a PTR record mapping your public mail server IP to its FQDN (e.g. mail.company.com). You cannot create PTR records yourself through your domain registrar — they're managed by the IP address owner. Once created, confirm it with Resolve-DnsName -Type PTR <your-IP> or a tool like MXToolbox. Also make sure your mail server's EHLO greeting matches that FQDN exactly.

Enable DNSSEC for zones that need it to protect against cache poisoning (DNS responses are cryptographically signed). Enable Response Rate Limiting (RRL) to prevent your DNS servers from being used as amplification targets in DDoS attacks. Use DNS Socket Pool and Cache Locking (both available in Windows Server) to reduce cache poisoning risk. For high availability, deploy Anycast DNS or at minimum ensure geographic distribution of DNS servers so no single datacenter failure takes down name resolution. Restrict zone transfers to authorized secondary servers only. Implement monitoring and alerting for unusual query volumes or unexpected zone changes — DNS hijacking is often quiet and can go unnoticed without visibility.

There are two main options. The quick option is to use per-user MFA — enable it manually for each admin account in the Microsoft 365 admin center. The better option is to create a Conditional Access policy targeting directory roles (e.g. Global Administrator, Exchange Administrator) that requires MFA for those role holders. Conditional Access gives you more control — you can scope it to specific apps, exclude trusted locations, and add device compliance requirements later without touching user accounts one by one.

Check these in order: (1) Is the user's account in the SSPR-enabled group? SSPR may only be enabled for a specific group, not all users. (2) Has the user registered their authentication methods (phone, email, authenticator app)? If they never completed SSPR registration, they won't have a method to verify their identity. (3) Are the required authentication methods (e.g. phone + email) actually set up in the user's profile? (4) If this is a hybrid environment, is password writeback configured and healthy? SSPR in the cloud still needs to write the new password back to on-premises AD. (5) Check if a Conditional Access policy is blocking the SSPR flow.

This is almost certainly a Conditional Access policy. Go to the Entra ID sign-in logs and filter by that user — find the failed sign-in and look at the Conditional Access tab. It will show exactly which policy blocked the request and which condition wasn't met. The two most common causes: (1) The CA policy requires a compliant device (Intune-enrolled), and the personal iPhone hasn't been enrolled. Fix: have the user enroll their phone in Intune Company Portal, then ensure it shows as compliant before retrying. (2) The policy requires a Hybrid Azure AD Joined device, which personal phones can never satisfy — in this case the policy needs a different condition for mobile (e.g. "require approved client app" instead of device compliance, using Outlook for iOS/Android which enforces app protection policies). Always read the sign-in log first to confirm the exact failure reason before making changes.

Before touching any production policy: (1) Create break-glass accounts — two emergency admin accounts that are excluded from all Conditional Access policies, have strong passwords stored securely offline, and are monitored with alerts for any sign-in. These are your recovery path if a policy locks everyone out. (2) Create the CA policy in Report-Only mode first — it evaluates sign-ins and logs what would have happened without actually enforcing anything. Review the sign-in logs for at least a week to check for false positives. (3) Enable the policy for a pilot group of admin accounts you can test with. (4) After validation, expand to all admin roles during a maintenance window. (5) Monitor sign-in failure rates in Azure Monitor or the Entra ID sign-in logs after full rollout.

Enable Privileged Identity Management (PIM) in Entra ID (requires Entra ID P2 license). Convert the existing permanent role assignments to Eligible assignments — the users no longer have the role standing, they have to explicitly request activation. Configure the activation settings: require MFA on activation, require a justification/ticket number, set a maximum activation duration (e.g. 4 hours), and optionally require approval from another admin. Set up access reviews on a quarterly basis so role eligibility is reviewed periodically and removed if no longer needed. Configure alerts to notify when roles are activated, so the security team has visibility.

In hybrid identity, the on-premises UPN must match what Azure AD knows the user by. When UPN and email domain differ, sign-in breaks. Fix in this order: (1) In Microsoft 365 Admin Center, add and verify newcompany.com as a custom domain first — do this before changing anything on-prem. If you update UPNs before the domain is verified, users will be assigned an unverified domain and get locked out worse. (2) Once newcompany.com is verified, update the UPN suffix for affected users in on-premises AD. You can do this individually in ADUC, or bulk-update with PowerShell: Get-ADUser -Filter * | Set-ADUser -UserPrincipalName "$($_.SamAccountName)@newcompany.com". (3) Run a delta sync: Start-ADSyncSyncCycle -PolicyType Delta. (4) Test with one user before doing the bulk change — confirm they can sign into M365 with the new UPN. Also update the primary SMTP address in Exchange/Exchange Online to match so email routing is consistent.

A standard MSP tenant baseline includes: Conditional Access: a baseline policy for admins (MFA + compliant device) and a policy for standard users (MFA for risky sign-ins + risk-based enforcement via Identity Protection). Passwordless: push toward FIDO2 keys or Windows Hello for Business for admin accounts. Break-glass: two emergency accounts per tenant, excluded from CA, monitored for any use. Identity Protection: configure risk policies (User Risk and Sign-in Risk) to block or MFA-challenge high-risk sign-ins automatically. B2B Governance: limit who can invite external guests. SCIM Provisioning: automate user lifecycle through HR connectors where possible to reduce orphaned accounts. Access Reviews: quarterly review of group memberships, privileged roles, and guest users. Monitoring: Sentinel or equivalent aggregating sign-in logs from all tenants into a single SIEM, with alerts for break-glass use, bulk deletions, and MFA bypass events.

Run the Hybrid Configuration Wizard (HCW) from the on-premises Exchange server — this automates most of the connector setup between on-prem and Exchange Online. After running the HCW, validate: (1) Send and receive connectors were created correctly in both environments. (2) Accepted domains are configured in Exchange Online. (3) End-to-end mail flow works in both directions — send a test email on-prem to EXO and vice versa. (4) Schedule migration batches during off-hours to minimize user impact. (5) Run health checks using Test-MigrationServerAvailability and the Microsoft Remote Connectivity Analyzer.

Point the domain's MX record to the Barracuda gateway (not directly to Exchange Online). Configure the Barracuda to relay accepted mail to Exchange Online (either to the tenant's *.mail.protection.outlook.com address). On the Exchange Online side, create an inbound connector that expects mail from the Barracuda's IP range and requires TLS. For SPF, update the DNS TXT record to include: both the Barracuda sending range and Exchange Online (include:spf.protection.outlook.com). Validate that domain ownership is verified in Microsoft 365 and test inbound and outbound mail flow end-to-end.

Read the NDR error code first — the code tells you exactly what failed. "550 5.7.1" can mean several things: (1) Outbound connector misconfigured — if mail is supposed to route through a gateway, check that the outbound connector in Exchange Online is scoped to the right domains and the gateway is accepting relay from Exchange Online's IP range. (2) SPF over-limit — if your SPF record has more than 10 DNS lookups, some receiving servers reject outbound email. Check your SPF record with MXToolbox. (3) Sending domain on a blocklist — your sending IP may have been added to a spam blacklist. Check with MXToolbox Blacklist Checker and consider submitting a delisting request. (4) TLS enforcement mismatch — if your outbound connector requires TLS but the recipient's server doesn't support it, delivery fails. Check the NDR details for a TLS negotiation error. Always start by reading the full NDR and using the Message Trace in the Exchange Admin Center to see exactly where in the delivery chain it failed.

Free/busy in hybrid relies on OAuth authentication between Exchange Online and on-premises Exchange. Check: (1) OAuth is configured and healthy — use Test-OAuthConnectivity in the Exchange Management Shell. (2) The on-premises Exchange Web Services (EWS) endpoint is reachable from Exchange Online — it must be publicly accessible. (3) The Availability Address Space is configured correctly in both environments. (4) Autodiscover is pointing correctly for both on-prem and cloud users (see Q12). (5) The MRS Proxy (Mailbox Replication Service) is healthy — failures here affect both mailbox moves and free/busy. Check virtual directory configurations and confirm the correct authentication methods are set on EWS virtual directory.

Three layers of email authentication: (1) SPF — already covered, publish a DNS TXT record listing all authorized sending IPs. (2) DKIM — enable DKIM signing in Exchange Online (or Defender for Office 365). This adds a cryptographic signature to outbound emails so recipients can verify the email genuinely came from your domain. (3) DMARC — publish a DMARC DNS record that tells receiving servers what to do when SPF/DKIM fail. Start with p=none (monitoring only) to identify legitimate mail sources that might break, then move to p=quarantine and finally p=reject once confident. Also configure Safe Links and Safe Attachments in Microsoft Defender for Office 365 for inbound protection against phishing and malicious attachments.

(1) First, use the Message Trace in the Exchange Admin Center to confirm the rule is what's catching the emails. The trace will show the rule name and action taken. (2) Check the quarantine in the Microsoft Defender portal — release a few specific messages to the recipient as a short-term fix so they receive their invoices today. (3) Review the transport rule itself — "subject or body contains 'invoice'" is too broad. Narrow it: instead of matching any email with that word, add an exception for verified vendor domains (e.g. SenderDomainIs: trustedvendor.com) or require the condition to match both a keyword AND an attachment type like .exe or .zip (malicious "invoice" emails typically arrive with harmful attachments). (4) Check if the legitimate emails pass SPF/DKIM/DMARC — failed authentication is often a contributing trigger. (5) Document the rule change in the change record before applying it.

Cutover migration moves all mailboxes at once over a weekend. It's suitable for small organizations (under ~150 mailboxes), where downtime can be planned and coexistence is not needed. It's simpler — you don't need an on-premises Exchange server afterwards. Hybrid migration is better for larger organizations or where there are compliance requirements, legal holds, or journaling in place — you can migrate in batches over weeks or months with zero user disruption because email flows to both environments simultaneously. With 150 mailboxes, either could work, but if there are compliance holds, eDiscovery requirements, or active litigation, hybrid is safer because it lets you migrate in staged batches and validate each group before cutting over. Also ensure SSL certificates and DNS are cleaned up post-migration — expired on-prem certs can break hybrid mail flow silently.

First, determine if this is a network issue or a device issue. (1) Try to reach the device from multiple points — if your monitoring server itself has a network issue, the alert may be a false positive. (2) Check if the HA (High Availability) partner took over — most enterprise firewalls run in active/passive HA. If failover happened, the secondary device is now primary and traffic is flowing. (3) Check with the ISP or upstream provider for any known outages. (4) Attempt to reach the device on its out-of-band management interface if available. (5) Escalate to the client with clear evidence: timestamp, what's unreachable, and what is still working. Document everything in the ticket.

Go to the firewall's traffic logs and filter by the user's source IP and the destination IP/port. Look for sessions that show "Deny" or "Drop" — this tells you which policy rule matched and blocked the traffic. In Palo Alto, use the Session Browser (Monitor > Session Browser) and run a traffic filter. In FortiGate, check the Forward Traffic log. If you need to go deeper, use packet capture on the firewall to see exactly what's arriving and what's being dropped. Also check rule hit counters — a rule with zero hits that you expected to match tells you the traffic isn't reaching it, which is a different problem (routing, or wrong source/dest in the rule).

(1) Log into the Meraki Dashboard — this is your first tool. Check the branch MX's uplink status. The dashboard shows whether each WAN interface is up or down, current utilization, and any uplink failover events in the event log. (2) If the primary WAN is shown as down, check whether a secondary uplink (4G failover or secondary ISP) has taken over. If failover happened, users should have limited internet — confirm they have connectivity at all. (3) Before calling the ISP, test from inside: can devices ping the MX gateway IP? Can they ping 8.8.8.8? If 8.8.8.8 pings but DNS names don't resolve, the issue is DNS, not internet access — that's a different fix (check the DNS server setting in the Meraki DHCP scope). (4) If the WAN is down and no failover, call the ISP with the circuit ID and request a reference number. (5) Update the ticket with timestamp, what's down, what's working, and the ISP case number.

(1) Confirm a change window is approved and you have a rollback plan documented. (2) Create a NAT rule that translates the public IP (or port) to the internal server IP. (3) Create a security policy rule that allows traffic from the Untrust zone to the DMZ or Trust zone on the required ports (e.g. TCP 443) — be specific with service objects, don't use "any." (4) Attach an appropriate security profile (threat prevention, URL filtering) to the rule so inbound web traffic is inspected. (5) Commit the changes. (6) Validate with a packet capture from outside (or from a test host) that the connection reaches the app. (7) Verify the rollback plan is ready — if the app breaks something, you should be able to reverse within minutes.

On both sides (HQ and branch): (1) Configure IKE Phase 1 — peer IP addresses, authentication (PSK or certificate), encryption algorithm (e.g. AES-256), Diffie-Hellman group (prefer Group 14+), lifetime. (2) Configure IPSec Phase 2 — encryption and hashing algorithms, proxy IDs (the local and remote subnet pairs that will be encrypted). (3) Configure Dead Peer Detection (DPD) so the tunnel re-establishes automatically if one side goes down. (4) Set up routing — either a static route pointing through the VPN interface, or BGP if you need dynamic routing. (5) Enable NAT-T (NAT Traversal) if either side is behind a NAT device. (6) After configuration, bring up the tunnel and verify with flow monitor or traffic logs that encrypted traffic is passing. Document both sides including PSK handling (store securely, not in plain text).

The most common cause of periodic drops is a Phase 2 (IPSec SA) lifetime mismatch — if HQ has Phase 2 lifetime set to 3600 seconds and the branch is set to 28800 seconds, the tunnel tears down when HQ tries to renegotiate and the branch rejects it (or vice versa). Both ends must have matching Phase 1 and Phase 2 lifetimes. Check and align the settings on both firewalls. Second cause: Dead Peer Detection (DPD) not configured or aggressive mode off — DPD monitors whether the other side is alive and should automatically re-establish the tunnel if a keepalive fails. Enable DPD on both ends. Third cause: NAT-T not enabled if either device is behind a NAT device — without NAT Traversal, keepalive packets may be dropped by an upstream NAT device. Fourth: check if the ISP resets idle sessions on a timer — some ISPs clear idle connections, which looks like a 4-6 hour drop. Ask the ISP. Fix by sending continuous low-bandwidth traffic (like DPD keepalives) to keep the session active.

Define security zones at each site: at minimum — Untrust (internet), DMZ (public-facing services), Trust (servers), User (workstations), and IoT (isolated). Use VLANs to enforce segmentation at Layer 2, with firewall rules controlling inter-VLAN traffic (Layer 3). Apply least-privilege policies — users can reach specific server ports they need, not "any to any." On Palo Alto, use App-ID to identify applications and User-ID to apply policies per user group. Attach threat prevention profiles to all rules to inspect traffic even within trusted zones. For critical applications, enable SSL decryption so encrypted threats don't bypass inspection. Deploy firewalls in HA pairs at each site to eliminate single points of failure. All changes must go through a CAB (Change Advisory Board) process with documented testing and rollback steps.

Follow VMware's documented certificate replacement procedure for your version — vCenter 7/8 uses the Certificate Manager tool (vecs-cli / certificate-manager). Key steps: (1) Generate a new Certificate Signing Request (CSR) with the correct CN and SAN entries (both the FQDN and any short names used to reach vCenter). (2) Sign it with your CA (either ADCS or a public CA). (3) Import the new certificate along with the full chain (intermediate + root CAs). (4) Restart vCenter services after the import. (5) Verify the web client loads without certificate warnings and confirm SAN entries match all access methods. (6) Check that dependent integrations (backup software, monitoring, ESXi hosts trusting vCenter) still work — they may have pinned the old certificate thumbprint.

(1) If the VM has snapshots, delete or consolidate them first — you cannot expand a VMDK with an active snapshot in most configurations. (2) In vCenter/Hyper-V, increase the virtual disk size (e.g. from 100 GB to 150 GB). (3) Inside the Windows VM, open Disk Management and rescan disks. The unallocated space will appear next to the existing volume. (4) Right-click the C: volume and choose Extend Volume — this grows the filesystem to use the new space. If Disk Management doesn't show the option, use diskpart with the extend command. (5) Verify the new drive size in File Explorer.

Check the Tasks and Events tab on the VM in vCenter — the error message there tells you exactly why it failed. Common reasons and fixes: (1) Insufficient CPU or memory on the host — the error will say "Unable to allocate X MHz" or "Unable to reserve X MB." Check the host's resource utilization. Try powering on from a different host, or if using DRS, put the cluster in fully automated mode and let it find a host. (2) Datastore not accessible — if the VMDK is on a datastore that's disconnected, unmounted, or out of space, the VM won't start. Check datastore connectivity in the Storage view. (3) CPU or memory reservation too high — if the VM has a hard reservation set higher than what's currently free on the host, it refuses to start. Reduce the reservation under VM Settings → Resources. (4) Virtual hardware version incompatible — if the VM was moved from a newer version of vCenter/ESXi, its virtual hardware version may be too high for the target host. Upgrade the host or move the VM to a compatible host.

Enable vSphere HA on the cluster for automatic VM restart on host failure. Prerequisites: shared storage (NFS, iSCSI, vSAN) accessible from all hosts, and at least two management network paths for heartbeating. Configure Admission Control so the cluster always reserves enough capacity to restart VMs from the largest host that could fail. Enable DRS (Distributed Resource Scheduler) to automatically balance VMs across hosts using vMotion. Set DRS to Fully Automated mode for continuous rebalancing, or Partially Automated if you want to approve migrations manually. Test HA by simulating a host failure in a maintenance window and confirming VMs restart on other hosts within the expected time.

(1) Generate a CSR (Certificate Signing Request) from vCenter with the correct CN and SANs. (2) Submit the CSR to ADCS — either through the web enrollment interface (https://CA/certsrv) or via command line with certreq. Use a Web Server template or a custom template that allows SAN entries. (3) Approve the request and download the signed certificate plus the full chain (intermediate CA and root CA). (4) Import the certificate into vCenter and restart services. (5) Ensure all machines that need to trust vCenter have the ADCS root CA in their Trusted Root store — on domain-joined machines, this is handled automatically through Group Policy. (6) For automation, consider scripting renewal using certreq or a PKI management tool so certificates don't expire silently.

Click the vMotion task error for specifics — the compatibility check error message is specific. Common causes and fixes: (1) CPU incompatibility — the destination host has a different CPU generation. Check if EVC (Enhanced vMotion Compatibility) mode is enabled on the cluster. If not, enable EVC to standardize the CPU instruction set presented to all VMs (requires powered-off VMs to take effect, but EVC ensures future vMotions work). Short term: migrate to a host of the same CPU generation. (2) Port group not present on destination host — the VM uses a network port group (e.g. "VLAN-Finance") that isn't configured on the target host. Add the port group to the target host's vSwitch. (3) ISO or CD-ROM connected — if a VM has an ISO file attached from a local datastore (not shared), vMotion can't move it. Disconnect the CD drive and retry. (4) Shared storage missing — the VM's datastore isn't mounted on the destination host. Rescan storage on the destination host.

Separate management infrastructure (vCenter, NSX Manager) into a dedicated management cluster so a production failure doesn't take down the tools you need to respond. Use fault domains within clusters to ensure VMs are distributed across physical failure zones (different racks, power feeds). For the 1-hour RTO, configure vSphere HA for automatic recovery. For 4-hour RPO, use vSphere Replication with replication interval set to under 4 hours, or a third-party tool like Veeam replicating to a DR site. For planned DR, consider Site Recovery Manager (SRM) — it orchestrates the entire failover runbook including network remapping and dependency ordering. Apply lifecycle and patch management via vSphere Lifecycle Manager (vLCM) to maintain consistency. Document the RTO/RPO targets, test failover quarterly, and produce audit evidence of the tests.

(1) Open the backup software (e.g. Veeam) and find the specific job. Read the error message — it usually tells you exactly what failed. Common causes: repository out of space, transport mode failure, network timeout, VSS error on the source. (2) Check the backup repository storage — if it's full, clean up old restore points or expand storage before retrying. (3) Check the transport mode (hotadd, network, direct) and whether the proxy has connectivity to the VM host. (4) Capture the error code and full log for the ticket. (5) Manually trigger a retry run and monitor it. (6) If this is a recurring failure, escalate to L2 — don't just keep retrying the same broken job. (7) Note the retention impact — if the last successful backup is now older than the SLA allows, notify the client immediately.

(1) First, check that Changed Block Tracking (CBT) is enabled on all VMware VMs. If CBT is disabled, every "incremental" backup is actually a full backup — this alone can make a job take 10x longer than expected. To verify, look at the data transferred in recent job sessions — if it matches the total VM size every night, CBT isn't working. Reset CBT if needed. (2) Check the backup proxy resources. If the proxy VM only has 2 vCPUs and is processing 50 VMs, it's the bottleneck. Add more proxy VMs to parallelize backup processing. (3) Review the transport mode — if it's using Network mode (going over LAN), switch to HotAdd (via datastore snapshots) or SAN Direct for much faster throughput. (4) Split the single large job into multiple smaller jobs that can run in parallel — e.g. one job for critical servers, one for file servers, one for dev machines, each running simultaneously. (5) Consider setting a soft job time limit: if the job hasn't finished by 5:30am, it saves what it has and resumes next cycle with a "retry missed VMs" setting.

File-level restore: Open the backup in Veeam, browse to a specific folder, restore 10–15 random files to an alternate location (not the original, to avoid overwriting production data), then compare checksums or open the files to confirm they're intact. Full VM restore: Restore the VM to an isolated virtual network (no production network access) — this is critical so the test VM doesn't conflict with the live system (duplicate IPs, duplicate AD machine account). Start the VM and verify: it boots, the OS is healthy, the application loads, and data is current to the backup timestamp. Document the results: restore time, backup age, what was validated, and any issues found. Use Veeam SureBackup (or equivalent) for automated validation — it spins up the VM in an isolated lab and runs application-specific tests automatically.

Agentless backups use the VMware VADP (vSphere Storage APIs for Data Protection) — they communicate directly with the hypervisor and snapshot the VM at the storage level without needing software installed inside each guest OS. Benefits: (1) No agent to install, update, or license on every VM — this simplifies management and eliminates agent-related failures. (2) Leverages Changed Block Tracking (CBT) to only back up changed data blocks since the last backup, making incremental backups very fast. (3) Multiple transport modes (hotadd, network, SAN direct) allow optimizing performance based on infrastructure. Agentless also handles VMs that you can't or don't want to touch from inside (e.g., appliances, locked-down systems). For application-consistent backups, VMware Tools must be installed so the hypervisor can trigger VSS quiescing inside the guest.

First 15 minutes (contain): (1) Isolate affected systems from the network immediately — disable switch ports, disconnect network cables, or put VMs on an isolated VLAN. Do not shut down VMs yet; memory may hold forensic evidence (encryption keys, running processes). (2) Identify the blast radius — which servers are encrypted, are domain controllers affected, and most critically, is the backup server or repository encrypted? Ransomware specifically targets backup systems. (3) Check whether backups are intact — if the backup server is also encrypted, you may be restoring from offsite or cloud copies. Recovery: (4) Determine the last known clean backup timestamp — before the infection, not after. (5) Restore to an isolated environment first, validate data integrity and that no encrypted files made it into the restore point. (6) Reset all credentials (domain admin, service accounts, local admin) before reconnecting any system — if you don't, the attacker's persistence (which is how they got in) will give them access again immediately. (7) Rebuild affected endpoints from clean OS images rather than restoring from potentially infected backups. (8) Document everything for the insurance claim and incident report.

A DR test plan includes: (1) Scope definition — which systems are in scope this quarter (rotate through all critical systems across quarters). (2) Dependencies map — document what each system depends on (AD, DNS, SQL, storage) so the failover sequence is correct. (3) Failover sequence — what restores first (infrastructure, then apps, then end-user services). (4) Network mapping — confirm DR site IP addresses, firewall rules, and DNS entries are pre-configured. (5) Test execution — perform the failover in an isolated environment where possible, measure actual RTO (time to recovery) and RPO (data loss window). (6) Rollback plan — how to return to primary site. (7) Audit artifacts — screenshots, logs, sign-off sheet documenting what was tested and the results. (8) Lessons learned meeting — what failed, what was slower than expected, what needs updating in the runbook. (9) Communication plan — who is notified during and after the test.

An App Registration is the definition of an application — it lives in your home tenant and defines what the app is, what permissions it requests, and how it authenticates. Think of it as the blueprint. An Enterprise Application (Service Principal) is the instance of that app as it appears in a specific tenant — it's what you configure for user assignment, SSO settings, and provisioning. If you build your own app, you create an App Registration. When you add a third-party SaaS app (like Salesforce, Slack, or DocuSign) from the gallery, Entra ID creates an Enterprise Application in your tenant that references the vendor's registration. Permissions, consent grants, and access assignments are managed on the Enterprise Application object.

AADSTS error codes are your friend — each one tells you exactly what failed. AADSTS700016 means the application ID in the SSO request doesn't match any App Registration or Enterprise Application in your tenant. This can happen for several reasons: (1) The app was registered in a different tenant (e.g. a dev tenant), and the production tenant doesn't have it. Fix: check which tenant the user is in and whether the app's Enterprise Application exists there. (2) The Application ID (Client ID) in the app's SSO configuration is wrong — a typo or wrong value was entered. Compare the Client ID in the app's vendor SSO settings against the actual App Registration in Entra ID. (3) The Enterprise Application was accidentally deleted — check the Entra ID audit log for deletion events and restore it if within the 30-day soft-delete window. When troubleshooting any AADSTS error, look it up in the Microsoft error code reference — it will tell you the specific failure and recommended fix.

(1) Add the app from the Entra ID App Gallery (if available) or register it manually as an Enterprise Application. (2) Choose the SSO method — OIDC (OpenID Connect, modern) or SAML (legacy, still very common). (3) For SAML: configure the Entity ID and Reply URL (ACS URL) provided by the vendor. Map the claims the app expects (usually email and display name at minimum) to Entra ID attributes. (4) Set redirect URIs and generate a client secret or certificate for the app to authenticate. (5) Enable SCIM provisioning if the app supports it — this automatically creates/deactivates user accounts in the app based on Entra ID group membership. (6) Test SSO with a dedicated test account before rolling out to all users. (7) Document the configuration including client secret expiry date for renewal tracking.

This is a serious security concern — OAuth consent phishing is a known attack vector. Immediate actions: (1) Review the app in Entra ID Enterprise Applications and check what permissions were granted. (2) Revoke the app's permissions and remove it if it's not legitimate. (3) Review audit logs to see what data the app may have accessed. Prevention: (1) Disable user consent for apps entirely in Entra ID settings (Entra ID → Enterprise Applications → Consent and Permissions → set to "Do not allow user consent"). (2) Enable the Admin Consent Workflow — users can request access, but an admin must approve it. (3) Periodically review all consented app permissions using the Entra ID access reviews or Defender for Cloud Apps (which can detect risky OAuth apps). (4) Apply Conditional Access to restrict app sessions for high-risk applications.

The pattern "worked Friday, broken Monday, no changes made" strongly points to an expired client secret. App Registration secrets have an expiry date (1 or 2 years when created) and when they expire, SSO stops working silently with no automatic warning or email. Verify: go to Entra ID → App Registrations → find the Salesforce app → Certificates & Secrets. Look at the expiry date on each secret. If one expired over the weekend, that's your answer. Fix: generate a new secret, immediately update the Salesforce SSO configuration with the new secret value (wherever the vendor stores it — usually in their SSO/IdP settings), and test. Prevention: export all App Registration secrets to a spreadsheet or use Microsoft Graph API to build an alert that fires when any secret is within 60 days of expiry. Consider switching to certificate-based authentication where supported — certificates have visible expiry management and are harder to accidentally overlook.

Standardize the SSO pattern across all tenants: SAML or OIDC, claim mappings, and provisioning configuration templates. Apply Conditional Access policies to app sessions (require compliant device and MFA for sensitive apps). Run quarterly Access Reviews to catch users who still have app access after leaving a project or role. Use Defender for Cloud Apps to detect risky OAuth consent grants across tenants. Manage all client secret and certificate expiry dates in a central tracker with renewal alerts at 60 and 30 days before expiry — this is one of the most common MSP failure points. Define a formal offboarding procedure for each app — when a client terminates, revoke the app's permissions and remove the enterprise application from the tenant. Document key rotation SOPs and ensure they are followed consistently.

AVD makes sense when: (1) The workforce uses shared or pooled desktops — AVD supports multiple users sharing one session host, which significantly reduces cost. (2) User profiles need to roam — FSLogix profile containers attach the user's profile from Azure Files storage, so the user gets the same desktop experience regardless of which session host they land on. (3) Application delivery is needed — you can publish specific apps (like a legacy line-of-business tool) without giving users a full desktop. (4) Remote access security is a concern — data never leaves Azure, reducing data loss risk from physical devices being stolen. (5) The workforce size varies — AVD scales up during busy periods and down overnight, reducing costs compared to dedicated hardware for every user.

(1) Since other users on the same pool are fine, this is user-specific, not host-wide — point your focus at the user's profile. (2) Check the Windows Event Log on the session host under Applications and Services Logs → Microsoft → Windows → User Profile Service for Event IDs 1500, 1502, 1508, 1509. These will tell you if the profile failed to load. (3) The most common cause: FSLogix is trying to attach the user's VHD but timing out or failing. Check the FSLogix log at Event Viewer → Applications and Services Logs → Microsoft → FSLogix → Profiles. (4) Try temporarily adding the user to the FSLogix exclusion group (by default: FSLogix Profile Exclude List) — this makes them log in with a local temporary profile. If the black screen goes away, the VHD is the problem. (5) Check if the VHD file is locked — if the user's last session crashed without cleanly unmounting, the VHD may still show as in use. You may need to reboot that session host to release the lock.

(1) Check the FSLogix event log on the session host (Event Viewer > Applications and Services Logs > Microsoft > FSLogix > Profiles) — the error here is almost always the root cause. (2) Verify the storage path configured in FSLogix (usually in the registry at HKLM\SOFTWARE\FSLogix\Profiles\VHDLocations) and confirm that path is reachable from the session host. (3) Test that the session host can actually connect to the Azure Files share using the correct credentials — the session host (or its managed identity) must have Storage File Data SMB Share Contributor permissions. (4) Check network latency to the storage account — high latency causes FSLogix to time out trying to attach the VHD. (5) Review FSLogix exclusions — certain user accounts (like admin accounts) may be excluded from FSLogix by design. (6) If the VHD file is locked (from a previous crashed session), it may need to be manually unlocked or the session host rebooted.

"Unavailable" in AVD means the AVD Agent on that session host is not checking in with the AVD service — it's not a VM-down problem. (1) Try to RDP directly into the unavailable session host from the Azure portal (use "Connect" → RDP, which bypasses the AVD broker). If you can connect, the VM is up but the agent is broken. (2) Once connected, check the AVD Agent and AVD Agent Bootloader services — if they're stopped, start them and check if the host comes back online in the portal. (3) Check the Windows Event Log → RemoteDesktopServices on the session host for agent registration errors. (4) Confirm outbound HTTPS (port 443) connectivity from the session host to AVD service URLs — a firewall rule or proxy may be blocking the agent's communication. (5) If the session host was recently updated (Windows Update or image rebuild), it may need to be re-registered to the host pool. Remove it from the pool and re-add it using the registration token. (6) Check that the session host's managed identity has the required role (Desktop Virtualization Session Host Operator) on the host pool resource.

Host Pools: Create pooled host pools (multiple users per VM) for standard knowledge workers. For users who need dedicated desktops (due to GPU requirements or app compatibility), create personal host pools. Session host sizing: Size VMs for a 4:1 to 6:1 users-per-core ratio depending on workload. For 500 users with standard Office workloads, consider D4s_v4 (4 vCPU, 16 GB RAM) or D8s_v4 hosts. Autoscale: Configure the AVD Autoscale plan — scale up session hosts before the morning peak (pre-warm schedule), scale down to minimum during off-hours. Drain and deallocate idle hosts to stop paying for them overnight. Golden Image: Create a custom VM image using Azure Image Builder or Intune, kept up to date with patches. Cost optimization: Use Azure Reserved Instances for baseline capacity (the minimum always-on hosts), Spot VMs for burst capacity if interruption is tolerable. Monitoring: Azure Monitor Workbooks for AVD, alert on user session count approaching capacity, and on FSLogix errors. Security: Apply Intune compliance policies to session hosts, enable Microsoft Defender for Endpoint, and apply a security baseline GPO.

(1) Reachability: Can you ping the server? If not, it may be powered off, crashed, or a firewall rule changed. Try Test-NetConnection -Port 3389 to check specifically if port 3389 is responding. (2) RDP service: If the server is reachable on ping but not port 3389, check if the Remote Desktop service is running on the server (if you can reach it another way — console, iLO, or another admin tool). (3) Group membership: Confirm the user is in the Remote Desktop Users local group on the server. A GPO refresh or group change could have removed them. (4) GPO restrictions: A new GPO might have blocked RDP access. Check gpresult /h for any deny policies. (5) Firewall: Check Windows Firewall on the server and any network firewall between the user and server — a rule may have been changed or a NAT entry expired. Document all findings in the ticket even if you resolve it quickly.

(1) RDP to the server and use WinDirStat or the built-in Get-ChildItem -Recurse | Sort-Object Length -Descending to find the largest files and folders. Don't run Disk Cleanup blindly before knowing what's consuming space. (2) Check the usual suspects in order: Windows Update cache (C:\Windows\SoftwareDistribution\Download — safe to clear after updates are installed), IIS/application logs (C:\inetpub\logs or custom app log paths — these can grow for months unnoticed), Windows crash dumps (C:\Windows\MEMORY.DMP and C:\Windows\Minidump\ — confirm no open investigations before deleting), print spooler files (C:\Windows\System32\spool\PRINTERS), and large user profiles or temp files on C:. (3) Run Disk Cleanup with "Clean up system files" to safely remove Windows Update remnants. (4) If the growth is continuous (e.g. application logs with no rotation), fix the root cause — configure log rotation, redirect logs to a data drive, or increase the drive size. (5) Document what was removed and how much space was recovered in the ticket.

Convert the resolution into a runbook (SOP — Standard Operating Procedure). A good runbook includes: (1) Pre-checks — how to confirm the issue matches this runbook before starting (e.g. "run repadmin /replsummary and confirm error 8524"). (2) Step-by-step resolution steps — written so an L1 tech can follow them without needing to understand every detail. (3) Impact statement — what is affected and what is not while this fix is being applied. (4) Rollback steps — what to do if a step makes things worse. (5) Validation steps — how to confirm the fix worked. (6) Escalation criteria — if step 3 doesn't resolve it, escalate to L2 immediately. Publish the runbook in your RMM or documentation system (ConnectWise, IT Glue, etc.), version-control it, assign an owner, and train L1 on it during the next team meeting.

(1) Classify servers into rings — not all servers get patched at the same time. Ring 0: test/dev servers, patched immediately after Patch Tuesday. Ring 1: non-critical production servers, patched 1 week later if Ring 0 had no issues. Ring 2: business-critical servers (domain controllers, SQL servers, core infrastructure), patched 2 weeks after Patch Tuesday in an approved maintenance window. (2) Always snapshot or backup before patching Ring 2 servers — if a patch breaks an application, you can roll back quickly. (3) Deploy patches via your RMM (NinjaRMM, ConnectWise, WSUS) rather than manually. Schedule reboots explicitly in the maintenance window, not immediately after patch install. (4) Automated health checks after patching: check services are running, event log for errors, application responds on its port. Your RMM should do this automatically. (5) Monthly compliance report to the client: what was patched, what's pending, and any patches deliberately deferred and why. (6) For the broken-application risk: test patches on Ring 0 (dev) first, and maintain an application-specific list of known-problematic patches that need testing before deployment.

(1) CAB review: Submit the change to a Change Advisory Board review. Document the business need, what the policy does, what users it affects, and the rollback procedure. Get sign-off from a senior engineer before any production change. (2) Staging: Apply the change to one or two low-risk test tenants first. (3) Report-Only mode first: In each tenant, enable the new CA policy in Report-Only mode and monitor sign-in logs for at least 48 hours. Look for any accounts that would have been blocked that shouldn't be. (4) Maintenance window: Schedule the actual enforcement change during a low-traffic window (e.g. Sunday 2am local time per tenant). (5) Monitor after rollout: Watch sign-in failure rates and help desk ticket volume for the 48 hours after enabling. Have the rollback ready (disable the policy). (6) Audit trail: Ensure every change is linked to a change record in your ticketing system. Post-implementation, link the change record to any incidents it caused — this builds the evidence that the CAB process is working and allows future audits.