I should also remind, that radare2[1], which is free and open source reverse engineering framework and can run also on Linux, *BSD and OS X, supports connection to WinDbg/KD[2]. So you can use your favourite operating system and debug Windows kernel in virtual machine or via debug cable.
It's also notable, that this year we're running Google Summer of Code[3] and hosting our own Radare Summer of Code[4]. And for the latter we're fundraising[5] the money to pay our accepted students.
I just started doing infosec coursework in my spare time. I go to cert courses and despite many people being better read than me, none of them know radare at all. I have a very introductory interest in the RE end, and the Radare2 book (which is a tome, full of so much free goodness) is just like a treasure troves of the infinite learning possibilities I will need to exploit it.
Seriously, thanks to the radare devs (if you are not one, xvilka, you talk like you are; in that case definitely thanks to you too). I am surprised in the era where people still people handsomely and brag about IDA Pro and scripts for it, I only learned about radare ironically from HN this year.
radare2 is a much needed alternative to all the software that require Windows. Full-blown radare2 with all the third party libraries has grown too big for my low resource computers, but I still keep a copy of ired as my preferred hex editor. Like netcat, one of those programs that is so useful I sometimes wonder why it's not part of the standard UNIX utilities. Best of luck with the SOC.
While I have no particular desire to use Windows for anything, I think it's very interesting to see the engineering tools used to support it. Though it's a proprietary system, it is nonetheless a large and successful one. I'm particularly interested in any information about the parts of the stack which collect OS and application crash dumps from a large distributed install base for remote post mortem analysis; hopefully there will be some articles about that too!
In another life I worked on Windows Error Reporting (WER) - the part of the stack that was responsible for collecting, analyzing and sending reports to Microsoft and even provide solutions to the user based on the analysis.
For usermode problems WER would do a first level analysis on the client by looking at the crashing thread's exception record (1) or in the case of a user mode hang it would do something much more sophisticated called wait chain traversal (2). It would then generate a signature for the issue and report it to the backend via a single (and cheap) http/https request. The backend would then increment a count for that particular signature and if needed, would ask the client for more information about the issue. Among other things it could ask the client to capture a dump of the crashing process and even a multiprocess dump in the case of hang that involved multiple processes (process A waiting for process B waiting for process C etc). The client would do the necessary collection, prompt the user if necessary (depending on the user's consent settings) and eventually send it off. The 2nd level collection was configurable via a internal web portal - teams could look up their top signatures and request for more information about it. They'd eventually get notified once the extra information was collected - for high frequency issues, the turnaround was very quick.
Kernel mode issues aka bluescreens were treated slightly differently. The client wasn't able to do first-level analysis on kernel mode crashes. Instead it would always send at atleast a minidump of the bluescreen to a backend service where a pool of servers would run the !analyze windbg extension in realtime on the minidump.
I work with Microsoft's support and I have fair idea about system used to automatically analyze dumps uploaded for all the support issues.Although i have not worked on the system itself ,I have contributed many rules for it in the past.This will not be as large as the OS or application crash dumps collected by the windows error reporting but I think the idea should be similar.
In the support scenarios, any dumps(can be up to multiple GBs of size as used by the process) collected and uploaded to the portal will be analyzed by a custom execution engine which is based on dbgeng.DLL .You can find a similar automated analysis with tool DebugDiag[1] which contains script for known issues such as crash,hang or high memory.
I have been doing break-fix for Windows, *nix, and Mac systems for a long time. I have worshipped at the altar of Russinovich and Sysinternals gods for as long. I have wanted to be able to sit with Process Explorer, Process Monitor, and Windbg and know everything that goes wrong with a system from weird driver buginess to Word crash for reasons ignortant users can show me happening once a week but I cannot find evidence. I know the basics, but I really want to go up to the advanced levels with this. After years, Windows is still a black box to me, more than it should be after reading more than basic material, and that upsets me.
I see two problems here:
1) Where is good comprehensive material on the topic?
I see pages like this, but no unified material and training. On the Sysinternals book, I caved and bought the Bible, Sysinternals v6 back when it was current. I routinely show it for geek cred to other IT friends; few even know it exists. I met some more advanced consultants a month ago and learned of the Aaron Margosis companion (https://www.amazon.com/Windows-Sysinternals-Administrators-R...). But for the advanced stuff for WinDbg I know there are tier 2 customer engineers who know stuff inside out who post blogs like this, but find no really thorough books/material/courses I would gladly pay for to learn the basics of this skillset. Advice? Suggestions? I wish Julia Evans (jvns.ca) counterpart exists for the beauty of Windows debugging tools, but I have not found her/him/it yet.
2) Does tooling exist beyond the Sysinternals and debuggers I keep missing because I am uninformed? Help!?
On the point of Julia Evans, tooling in Windows is good, but I have not found much beyond the blessed debuggers and the gap between these debuggers+Sysinternals stack and that of the raw power tools of the Unix landscape of late: dtrace, strace, ltrace which go into crazy levels of introspection. I looked after reading this and stumbled upon maybe Dr. Memory but have not had time to learn it. Do you have counterpoints or suggestions?
Again, I would love to learn fu like Julia Evans in the Unix space and begin a love affair with the Windows side of debugging tools, but the landscape is more confusing. I would seriously love to hear from you.
Thanks so much for pointing us to DebugDiag. I have never found this and I am checking it out now. Very exciting!
UPDATE: I am reading the Code Machine posts listed in another comment, but it seems that Advanced Windows Debugging book might be the start of what I am looking for. Would love to know if you have other suggestions.
Many troubleshooting steps in that blog also covers DebugDiag .DebugDiag is an amazing tool which can help you collect dumps automatically for crash,high memory,IIS slow response etc.
Perfview is built on the (Windows Event tracing) ETW infrastructure windows have .You can think of it as Lttng or perf_events linux equivalent but built into everything windows have. I think process monitor is also built on ETW. After debugging ( which is not required for lot of scenarios) the most powerfull thing in troubleshooting any issues in Windows is ETW traces IMHO.So understanding and analyzing ETW will help you troubleshoot a lot of issues.
Seriously, hitr, thanks for you taking the time to respond.
Also, the dead comment appears to have alluded to the 71-660. Once other things settle down, I am diving into Sysinternals and these books when I had put them aside.
> In the Hyper-V settings for the VM, set COM 1 to use a Named Pipe
I used to have an RS-232 cable in my office, for Kernel debugging. Before VM prevalence the debugged and the debugee would be two physical machines, linked with a physical cable over their COM ports. Once the VPC/VirtualServer came out with COM redirect and the debugger support for it, I never looked back.
It's also notable, that this year we're running Google Summer of Code[3] and hosting our own Radare Summer of Code[4]. And for the latter we're fundraising[5] the money to pay our accepted students.
[1] http://rada.re
[2] https://github.com/radare/radare2/blob/master/doc/windbg
[3] http://radare.today/posts/gsoc-midterm-2016/
[4] http://rada.re/rsoc
[5] https://www.indiegogo.com/projects/radare-summer-of-code-201...