Pre-Debug Recon (Invisible Setup) — Windows Fail-Proof Guide

This playbook collects high-signal telemetry without modifying code paths, lets you enable canary debug probes on demand, replays production-like traffic into ephemeral clones, and funnels all access through a stealth, audited jump host. Every step includes copy-ready commands, validations, cleanup, and troubleshooting.

Low overhead No code changes Admin rights required Windows 10/11

Quick Navigation

0) One-time Prep & Sanity Checks 1) Shadow Sessions (tcpdump/dumpcap, Procmon, WPR) 2) Canary Instrumentation (.NET feature flags) 3) Ephemeral Clones & Traffic Replay (HAR→k6, GoReplay, mitmproxy) 4) Stealth Entry (OpenSSH jump host + WEF audit trail) Operational Runbook & Stop-All References (official docs & guides)

0) One-time Prep & Sanity Checks

1. Create protected capture folder (rotate files, avoid permissions drift):

# Admin PowerShell
$CAP="C:\Captures"
New-Item -ItemType Directory -Force -Path $CAP | Out-Null
icacls $CAP /inheritance:r | Out-Null
icacls $CAP /grant:r "BUILTIN\Administrators:(OI)(CI)(F)" "NT AUTHORITY\LOCAL SERVICE:(OI)(CI)(M)" | Out-Null

2. Install Npcap & Wireshark tools (for dumpcap, the headless capture engine). During Npcap setup, check “Support loopback traffic”.

Official downloads:

• Npcap (packet capture driver): https://npcap.com/
• Wireshark (includes dumpcap.exe): https://www.wireshark.org/

3. Get Sysinternals Process Monitor and unzip someplace predictable (e.g., C:\Sysinternals).

Download: Process Monitor (Procmon)

4. Install Windows Performance Toolkit (WPR/WPA) via Windows ADK (only the “Windows Performance Toolkit” feature is needed).

Start menu → “Windows Performance Recorder” should exist after install. Docs: Windows Performance Toolkit

1) Shadow Sessions — Passive, Low-Impact Mirroring

1A) Packet capture with dumpcap (ring buffer, bounded disk)

Use Wireshark’s capture engine headlessly. We capture only what we need and rotate files automatically.

1. List interfaces and pick your target index:

"&C:\Program Files\Wireshark\dumpcap.exe" -D

2. Start ring-buffered capture (example: HTTPS to a single host; ~2GB cap):

$iface=1
$filter='tcp port 443 and host 10.0.0.25'
& "C:\Program Files\Wireshark\dumpcap.exe" `
  -i $iface -f $filter `
  -b filesize:100000 -b files:20 `
  -w "C:\Captures\shadow_%Y%m%d_%H%M%S.pcapng"

Validate: Files rotate in C:\Captures. CPU stays low; disk grows to ~2GB max then cycles.
Stop: Ctrl+C in the console that started it or close the window (capture stops).

1B) System call / registry / file I/O trace with Procmon (headless)

1. Open Procmon once (GUI) → add tight filters (e.g., Process Name is your service) → File → Save Configuration… to C:\Captures\procmon_config.pmc.
2. Run headless with a backing file (no RAM spikes):

$pm="C:\Sysinternals\Procmon64.exe"
& $pm /AcceptEula /Quiet /Minimized `
  /LoadConfig C:\Captures\procmon_config.pmc `
  /BackingFile C:\Captures\procmon.pml
# ... reproduce the issue window ...
& $pm /Terminate

Validate: procmon.pml grows; CPU remains moderate due to tight filters.
Analyze later: open PML in Procmon and export CSV if needed.

1C) Whole-system ETW trace with Windows Performance Recorder (WPR)

1. Start safe, file-mode profile:

wpr -start GeneralProfile -filemode

2. Reproduce for ≤10 minutes, then stop:

wpr -stop C:\Captures\perf_trace.etl

Analyze: open Windows Performance Analyzer (WPA) and load perf_trace.etl. Use CPU, Disk, File I/O, Registry, and Networking graphs.

2) Canary Instrumentation — Dormant Probes Behind Feature Flags (.NET)

Ship lightweight diagnostics disabled by default. Flip on per cohort or account; roll back instantly.

2A) Microsoft.FeatureManagement (ASP.NET Core)

1. Add package:

dotnet add package Microsoft.FeatureManagement.AspNetCore

2. Configure flag (appsettings.json):

{
  "FeatureManagement": { "DebugProbes": false }
}

3. Gate your probe endpoint:

using Microsoft.FeatureManagement;
[ApiController]
public class ProbeController : ControllerBase {
  private readonly IFeatureManager _features;
  public ProbeController(IFeatureManager f) => _features = f;

  [HttpGet("/_probe/ping")]
  public async Task<IActionResult> Ping() {
    if (await _features.IsEnabledAsync("DebugProbes")) {
      return Ok(new { ok=true, t=DateTimeOffset.UtcNow });
    }
    return NotFound();
  }
}

Flip on/off: Use Azure App Configuration (or any config provider) to toggle DebugProbes for a tiny user segment. Validate and disable promptly.

2B) Unleash (open-source flags) — optional

Run Unleash (Docker) and guard the same probe with client.IsEnabled("DebugProbes"). Use strategies (ByUserId, % Rollout) to define canaries.

3) Ephemeral Clones — Isolated Mirrors + Traffic Replay

Create a staging copy (containers/VM) with masked secrets. Test fixes using real request shapes.

3A) Browser HAR → k6 replay

1. In Edge/Chrome DevTools → Network → Export HAR as prod.har.
2. Convert to a k6 script:

npm i -g har-to-k6
har-to-k6 .\prod.har -o .\replay.js

3. Edit replay.js to point to your clone’s base URL, then run:

k6 run .\replay.js --vus 5 --duration 2m

3B) GoReplay — mirror live traffic to staging

Forward production traffic shapes to the clone (Windows supported with caveats):

.\gor.exe --input-raw :443 --output-http http://staging.example.local

3C) mitmproxy — record & replay saved flows

Record with mitmproxy (client trusts its cert), then client-side replay into clone for deterministic testing.

4) Stealth Entry — Jump Host (OpenSSH) + Read-Only Audit (WEF)

4A) Hardened Windows OpenSSH jump host

1. Install & start OpenSSH Server (Windows Optional Feature):

Add-WindowsCapability -Online -Name OpenSSH.Server~~~~0.0.1.0
Start-Service sshd
Set-Service sshd -StartupType Automatic

2. Key-based auth for admins (uses special Windows path):

ssh-keygen -t ed25519 -C "ops-admin"
# On jump host (Admin PS):
$ak="$env:ProgramData\ssh\administrators_authorized_keys"
New-Item -ItemType File -Force -Path $ak | Out-Null
# Paste your public key contents into $ak, then lock ACLs:
icacls $ak /inheritance:r /grant:r "Administrators:F" "SYSTEM:F"
Restart-Service sshd

3. Harden sshd_config (C:\ProgramData\ssh\sshd_config):

PasswordAuthentication no
PubkeyAuthentication yes
PermitRootLogin no

4. Restrict source IPs (office/VPN ranges only):

New-NetFirewallRule -DisplayName "SSH from Corp" -Direction Inbound `
  -Protocol TCP -LocalPort 22 -Action Allow -RemoteAddress 203.0.113.0/24

Azure: Prefer Azure Bastion (no public VM IPs). Enforce MFA with Conditional Access at the portal/management plane.

4B) “Read-only” audit via Windows Event Forwarding (WEF)

1. On the collector (central log server):

wecutil qc
# Event Viewer → Subscriptions → Create Subscription → Source-initiated
# Add Security, Sysmon (if present), OpenSSH/Operational channels

2. Push the SubscriptionManager value via GPO to sources (domain):

# Example string (adjust collector FQDN):
Server=http://collector.contoso.com:5985/wsman/SubscriptionManager/WEC,Refresh=10

Validate: On the collector, Subscriptions → Runtime Status shows “Active” and events arriving. Lock collector ACLs so devs cannot delete/alter logs.

Operational Runbook

1. Start dumpcap ring buffer (keeps running safely).
2. Start Procmon (headless, tight filters) and WPR (-filemode).
3. Reproduce for ≤10 minutes. Stop Procmon + WPR. Keep dumpcap if needed.
4. Enable DebugProbes flag for a tiny cohort (1–5%) → collect → disable.
5. Test fixes in the clone using k6 or GoReplay against the staging URL.
6. All admin access via jump host/Bastion; all logs land in WEF.

Stop-All (safe shutdown)

# Stop WPR if running:
wpr -cancel
# Stop Procmon if running with backing file:
& "C:\Sysinternals\Procmon64.exe" /Terminate
# Stop dumpcap: close the console that launched it, or Ctrl+C in that window

Troubleshooting (fast)

• dumpcap not capturing: Reinstall Npcap (check “Support loopback”), run PowerShell as Admin, confirm interface index with -D.
• Procmon too noisy: Re-save config with stricter filters (Process, Path prefixes), always use /BackingFile.
• WPR missing: Install Windows ADK → Windows Performance Toolkit; retry wpr -start GeneralProfile -filemode.
• SSH key auth fails: Ensure C:\ProgramData\ssh\administrators_authorized_keys exists, owned by SYSTEM, ACLs only Administrators and SYSTEM, restart sshd.
• WEF access denied: Run wecutil qc on collector, winrm qc on sources; confirm Event Log Readers membership and SubscriptionManager GPO.

References (curated, up-to-date)

• Npcap (driver): npcap.com • Release archive: /dist
• Wireshark dumpcap manual: dumpcap(1) • How-to: WSUG dumpcap
• Process Monitor (Sysinternals): Procmon
• Windows Performance Toolkit (WPR/WPA): WPT docs • WPA: analyzer
• .NET Feature Flags (Microsoft): reference • Quickstart: quickstart
• Unleash flags (.NET): tutorial • examples: examples
• k6 HAR converter: docs • har-to-k6
• GoReplay (Windows notes): Running on Windows • Wiki
• mitmproxy replay: Replay requests
• Fiddler AutoResponder: KB
• OpenSSH on Windows: install/first use: doc • server config: doc • key mgmt (admin file): doc
• Azure Bastion overview: doc
• Conditional Access (MFA): overview • Require MFA for Azure mgmt: policy
• WEF: wecutil • Source-initiated subscriptions: setup

Copyright © 2025. This document is self-contained and safe to print or share internally.