Author Archives: famellee

The Case of CoCreateInstance Failed Randomly with E_CLASS_NOT_REGISTERED

Last week I investigated an issue in the product where the calls made to CoCreateInstance failed randomly. There were some twists in the investigation; the result surprised almost everybody, and the process worth mentioning.

Since beginning the issue has been found for weeks, developers noticed that CoCreateInstance may randomly return error E_CLASS_NOT_REGISTERED, the “random” here means that it can happen on different code path, it can happen on arbitrary in-proc COM objects, it can happen on any thread, and it can recover automatically.

First, we ruled out the possibility of the COM objects were not registered, despite of what the error code says. We register these COM objects during product installation; they stay registered until product uninstall, so the issue must be somewhere else.

Next, we used Process Monitor captured trace log when we reproduced the issue. Process Monitor log shows that COM library was able locate the CLSID in registry, read out the value of InProcServer32 under it, and was able to open the DLL to map it into memory. However, it seems to be where COM library stopped, as there was no evidence from log that tells us the DllMain function from the DLL was ever called.
Here I started suspecting that somehow the NT loader actually failed to load the DLL. The best option of looking into NT loader issue is to enable show loader snaps with gflags. It is simple as run following command line and re-run the program within debugger:

gflags.exe -i program.exe +sls

After enabled show loader snaps and run the product again in WinDbg this time, I found following output from WinDbg:

1fac:133c @ 01643484 - LdrpInitializeNode - ERROR: Init routine 000000006CC81844 for DLL "C:\Program Files\Company\abc.dll" failed during DLL_PROCESS_ATTACH
1fac:133c @ 01643484 - LdrpProcessDetachNode - INFO: Uninitializing DLL "C:\Program Files\ Company\abc.dll" (Init routine: 000000006CC81844)
1fac:133c @ 01643484 - LdrpUnloadNode - INFO: Unmapping DLL "C:\Program Files\ Company\abc.dll"
1fac:133c @ 01643484 - LdrpLoadDllInternal - RETURN: Status: 0xc0000142
1fac:133c @ 01643484 - LdrLoadDll - RETURN: Status: 0xc0000142

Therefore, NT loader was able to map abc.dll into memory, but its initialize routine returned FALSE to loader, and loader in turned failed LoadLibrary. Normally in this scenario LoadLibrary will return error 1114, “A dynamic link library (DLL) initialization routine failed”.

That abc.dll was created with Visual Studio with default settings; it statically links to CRT with entry point of CRT function _DllMainCRTStartup. The next logic step is to use WinDbg to step through the function to find out where it failed.
Fortunately _DllMainCRTStartup is not complicated and I quickly narrowed the failure down to following function:

extern "C" bool __cdecl __vcrt_initialize_ptd()
{
    __vcrt_flsindex = __vcrt_FlsAlloc(&__vcrt_freefls);
    if (__vcrt_flsindex == FLS_OUT_OF_INDEXES)
    {
        return false;
    }

    if (!store_and_initialize_ptd(&__vcrt_startup_thread_ptd))
    {
        __vcrt_uninitialize_ptd();
        return false;
    }

    return true;
}

FlsAlloc returned FLS_OUT_OF_INDEXES, which in turned failed the CRT initialization. A little research shows that Fiber Local Storage has 128 slots per process; each DLL statically linked to CRT will take a slot during CRT initialization. There is possibility that if process has loaded too many DLLs and exhausted FLS slots, then new DLL statically linked to CRT will fail to load, and that is exactly what we ran into.

Since we knew what is happening, a solution is working in progress.

Convert number to literal string

Here is a little function that translates numbers like 123 into literal strings like “one hundred twenty three”, the result is in English only:

#include <string>
#include <map>
#include <stack>
#include <iostream>
#include <stdint.h>
using namespace std;

string num2str(uint64_t num) {
	using mapping_t = map<uint64_t, string>;
	static mapping_t mapping = { {0, "zero"},{1, "one"}, {2, "two"}, {3, "three"},{4, "four"},
								 {5, "five"},{6, "six"},{7, "seven"},{8, "eight"},{9, "nine"},
								 {10, "ten"}, {11, "eleven"}, {12, "twelve"}, {13, "thirteen"},
								 {14, "fourteen"}, {15, "fifteen"}, {16, "sixteen"}, {17, "seventeen"},
								 {18, "eighteen"}, {19, "nineteen"}, {20, "twenty"}, {30, "thirty"},
								 {40, "fourty"}, {50, "fifty"}, {60, "sixty"}, {70, "seventy"},
								 {80, "eighty"}, {90, "ninety"}, {100, "hundred"}, {1000, "thousand"},
								 {1000000, "million"}, {1000000000, "billion"}, { 1000000000000, "trillion"} };
	string str;
	if (num == 0) {
		str = mapping[0];
	}
	else {
		enum DATA_FLAGS {
			FLAG_REMAINING = 1,
			FLAG_UNIT
		};
		using stack_t = stack<uint64_t>;
		stack_t s;
		s.push(num);
		s.push(FLAG_REMAINING);
		while (!s.empty()) {
			uint64_t flag = s.top();
			s.pop();
			uint64_t data = s.top();
			s.pop();

			if (flag == FLAG_UNIT) {
				str += " ";
				str += mapping[data];
			} else {
				if (data <= 20) {
					if (!str.empty())
						str += " ";
					str += mapping[data];
				} else if (data < 100) {
					uint64_t tens = (data / 10) * 10;
					uint64_t digits = data % 10;
					if (!str.empty())
						str += " ";
					str += mapping[tens];
					if (digits > 0) {
						if (!str.empty())
							str += "-";
						str += mapping[digits];
					}
				}
				else {
					for (mapping_t::reverse_iterator it_last = mapping.rbegin(); it_last != mapping.rend(); ++it_last) {
						uint64_t base = it_last->first;
						if (data >= base) {
							uint64_t units = data / base;
							uint64_t remaining = data % base;
							s.push(remaining);
							s.push(FLAG_REMAINING);
							s.push(base);
							s.push(FLAG_UNIT);
							s.push(units);
							s.push(FLAG_REMAINING);
							break;
						}
					}
				}
			}
		}
	}

	return str;
}

Test code with a large number:

int main() {
	string str = num2str(1234567890123);
	cout << str.c_str() << endl;
    return 0;
}

Output:

one trillion two hundred thirty-four billion five hundred sixty-seven million eight hundred ninety thousand one hundred twenty-three

Read digital signatures from multi-signed files

Here is code example of how to use WinTrust to retrieve digital signatures and certificates from a file. It supports multi-signed and catalog-signed files.

What it does:

  • Use WinTrust API to enumerate all embedded digital signatures (including nested ones and timestamps) from the file and display the properties of each signature;
  • For each digital signature, establish the trust chain of the signing certiifcate, display propertie of each certificate in the trust chain;
  • Check if the file is catalog signed, and if yes, do the same to the catalog file: display properties of catalog file’s signatures and their certificates;

What it doesn’t:

  • Verify the signature
  • Verify the certificate
  • Verify the file against the signature

signature.cpp

#include "signature.h"

#include <vector>
#include <memory>
#include <map>
#include <sstream>
#include <functional>
#include <algorithm>
#include <ctime>
#include <iomanip>
#include <locale>
#include <codecvt>

#include <windows.h>
#include <wincrypt.h>
#include <softpub.h>
#include <mscat.h>
#pragma comment (lib, "crypt32.lib")
#pragma comment (lib, "wintrust.lib")

template<typename T, typename T::pointer invalid_value = nullptr> struct pointer_wrapper {
	pointer_wrapper(T& pointer) : _pointer(pointer), _raw_pointer(invalid_value) {}
	~pointer_wrapper() {
		if (_raw_pointer != invalid_value)
			_pointer.reset(_raw_pointer);
	}
	pointer_wrapper(pointer_wrapper&& _other) : _pointer(_other._pointer) {
		std::swap(_raw_pointer, _other._raw_pointer);
	}
	operator typename T::pointer*() && { return &_raw_pointer; }
	operator typename T::pointer&() && { return _raw_pointer; }

private:
	T& _pointer;
	typename T::pointer _raw_pointer;
	pointer_wrapper(pointer_wrapper::) = delete;
	pointer_wrapper operator = (const pointer_wrapper::) = delete;
	pointer_wrapper operator = (pointer_wrapper&&) = delete;
	static void* operator new (size_t) = delete;
	static void* operator new (size_t, void*) = delete;
	static void* operator new[](size_t) = delete;
	static void* operator new[] (size_t, void*) = delete;
};

template<typename T, typename T::pointer invalid_value = nullptr> pointer_wrapper<T, invalid_value> get_pointer_wrapper(T& _pointer) {
	return pointer_wrapper<T, invalid_value>(_pointer);
}

template <typename T, BOOL(__stdcall *f)(T), T null_t = 0> struct deleter_t {
	using pointer = T;
	void operator() (pointer value) {
		if (value != null_t)
			f(value);
	}
};

using HCRYPTMSG_t = std::unique_ptr<HCRYPTMSG, deleter_t<HCRYPTMSG, CryptMsgClose>>;
using CERT_CONTEXT_t = std::unique_ptr<CERT_CONTEXT, deleter_t<PCCERT_CONTEXT, CertFreeCertificateContext>>;
using HANDLE_t = std::unique_ptr<HANDLE, deleter_t<HANDLE, CloseHandle, INVALID_HANDLE_VALUE>>;

BOOL __stdcall CertCloseStore0(HCERTSTORE pcrypt_cert_store) {
	return CertCloseStore(pcrypt_cert_store, 0);
}
using HCERTSTORE_t = std::unique_ptr<HCERTSTORE, deleter_t<HCERTSTORE, CertCloseStore0>>;

BOOL __stdcall CertFreeCertificateChain0(PCCERT_CHAIN_CONTEXT pcrypt_cert_chain) {
	CertFreeCertificateChain(pcrypt_cert_chain);
	return true;
}
using CERT_CHAIN_CONTEXT_t = std::unique_ptr<const CERT_CHAIN_CONTEXT, deleter_t<PCCERT_CHAIN_CONTEXT, CertFreeCertificateChain0>>;

BOOL __stdcall CryptCATAdminReleaseContext0(HCATADMIN hcatadmin) {
	return CryptCATAdminReleaseContext(hcatadmin, 0);
}
using HCATADMIN_t = std::unique_ptr<HCATADMIN, deleter_t<HCATADMIN, CryptCATAdminReleaseContext0>>;

struct deleter_HCATINFO {
	using pointer = HCATINFO;
	deleter_HCATINFO(HCATADMIN catadmin) : _catadmin(catadmin) {}
	void operator() (pointer hcatinfo) {
		if (hcatinfo != 0)
			CryptCATAdminReleaseCatalogContext(_catadmin, hcatinfo, 0);
	}
	HCATADMIN _catadmin;
};
using HCATINFO_t = std::unique_ptr<HCATADMIN, deleter_HCATINFO>;

struct crypt_allocator : std::allocator<BYTE> {
	BYTE* allocate(std::size_t n) {
		return reinterpret_cast<BYTE*>(::CryptMemAlloc(static_cast<ULONG>(n)));
	}
	void deallocate(BYTE* p, std::size_t n) {
		if (p)
			::CryptMemFree(p);
	}
};

std::unique_ptr<tm> filetime_to_tm(const FILETIME* pft) {
	FILETIME ft_local = {};
	FileTimeToLocalFileTime(pft, &ft_local);
	SYSTEMTIME st = {};
	FileTimeToSystemTime(::ft_local, &st);
	std::unique_ptr<tm> t(std::make_unique<tm>());
	t->tm_year = st.wYear - 1900;
	t->tm_mon = st.wMonth - 1;
	t->tm_mday = st.wDay;
	t->tm_hour = st.wHour;
	t->tm_min = st.wMinute;
	t->tm_sec = st.wSecond;
	t->tm_isdst = -1;
	return std::move(t);
}

std::wstring string_from_filetime(const FILETIME* pft) {
	wchar_t str[128] = {};
	wcsftime(str, _countof(str), L"%c", filetime_to_tm(pft).get());
	return str;
}

std::wstring get_cert_field(PCCERT_CONTEXT pcrypt_cert, DWORD type, DWORD flags) {
	DWORD size = CertGetNameString(pcrypt_cert, type, flags, 0, 0, 0);
	if (size > 0) {
		std::vector<WCHAR, crypt_allocator> value(size + 1);
		size = CertGetNameString(pcrypt_cert, type, flags, 0, value.data(), static_cast<DWORD>(value.size()));
		return std::wstring(value.data());
	}
	return L"";
}

std::wstring get_cert_name(CERT_NAME_BLOB* pcrypt_blob, DWORD type) {
	DWORD size = CertNameToStr(X509_ASN_ENCODING, pcrypt_blob, type,  0, 0);
	if(size > 0) {
		std::vector<WCHAR, crypt_allocator> value(size + 1);
		size = CertNameToStr(X509_ASN_ENCODING, pcrypt_blob, type, value.data(), static_cast<DWORD>(value.size()));
		return std::wstring(value.data());
	}
	return L"";
}

std::wstring get_oid_info(DWORD type, void* p, DWORD groupid) {
	const PCCRYPT_OID_INFO pcrypt_oid_info = CryptFindOIDInfo(type, p, groupid);
	if (pcrypt_oid_info && pcrypt_oid_info->pwszName)
		return pcrypt_oid_info->pwszName;
	return L"";
}

std::wstring format_crypt_blob(const CRYPT_INTEGER_BLOB* pcrypt_blob, bool reverse = false, DWORD limit = 36) {
	std::wstringstream ss;
	ss << std::hex << std::setfill(L'0');
	DWORD size = min(limit - 3, pcrypt_blob->cbData);
	if (reverse) {
		for (DWORD n = size; n > 0; --n) {
			ss << std::setw(2) << pcrypt_blob->pbData[n - 1];
		}
	}
	else {
		for (DWORD n = 0; n < size; ++n) {
			ss << std::setw(2) << pcrypt_blob->pbData[n];
		}
	}
	if (size < pcrypt_blob->cbData)
		ss << L"...";
	return ss.str();
}

DWORD get_cert_property(const PCCERT_CONTEXT pcrypt_cert, DWORD property_id, std::vector<BYTE, crypt_allocator>& _buffer) {
	DWORD size = 0;
	if (!CertGetCertificateContextProperty(pcrypt_cert, property_id, 0, &size))
		return GetLastError();
	std::vector<BYTE, crypt_allocator> buffer(size);
	if (!CertGetCertificateContextProperty(pcrypt_cert, property_id, buffer.data(), &size))
		return GetLastError();
	std::swap(buffer, _buffer);
	return 0;
}

DWORD crypt_decode(LPCSTR decode_type, const std::vector<BYTE, crypt_allocator>& encoded, std::vector<BYTE, crypt_allocator>& decoded) {
	DWORD size = 0;
	if (!CryptDecodeObject(X509_ASN_ENCODING | PKCS_7_ASN_ENCODING, decode_type, encoded.data(), static_cast<DWORD>(encoded.size()), 0, 0, &size))
		return GetLastError();
	std::vector<BYTE, crypt_allocator> _decoded(size);
	if (!CryptDecodeObject(X509_ASN_ENCODING | PKCS_7_ASN_ENCODING, decode_type, encoded.data(), static_cast<DWORD>(encoded.size()), 0, _decoded.data(), &size))
		return GetLastError();
	std::swap(_decoded, decoded);
	return 0;
}

void decode_usage(CTL_USAGE* pcert_usage, pairs& usages) {
	std::wstring_convert<std::codecvt_utf8_utf16<wchar_t>> utf8_to_utf16;
	for (DWORD n = 0; n < pcert_usage->cUsageIdentifier; ++n) {
		const CCRYPT_OID_INFO* pcrypt_oidinfo = CryptFindOIDInfo(CRYPT_OID_INFO_OID_KEY, pcert_usage->rgpszUsageIdentifier[n], 0);
		std::wstring name, usage = utf8_to_utf16.from_bytes(pcert_usage->rgpszUsageIdentifier[n]);
		name = (pcrypt_oidinfo && pcrypt_oidinfo->pwszName) ? pcrypt_oidinfo->pwszName : usage;
		usages.push_back({ name, usage });
	}
}

DWORD get_cert_from_CERT_CONTEXT(const PCCERT_CONTEXT pcrypt_cert, CERT* pcert) {
	pcert->version = pcrypt_cert->pCertInfo->dwVersion;
	pcert->serial_number = format_crypt_blob(::pcrypt_cert->pCertInfo->SerialNumber, true);
	pcert->issuer = get_cert_field(pcrypt_cert, CERT_NAME_SIMPLE_DISPLAY_TYPE, CERT_NAME_ISSUER_FLAG);
	pcert->name = get_cert_field(pcrypt_cert, CERT_NAME_FRIENDLY_DISPLAY_TYPE, 0);
	pcert->valid_from = string_from_filetime(::pcrypt_cert->pCertInfo->NotBefore);
	pcert->valid_to = string_from_filetime(::pcrypt_cert->pCertInfo->NotAfter);
	pcert->subject = get_cert_name(::pcrypt_cert->pCertInfo->Subject, CERT_X500_NAME_STR | CERT_NAME_STR_REVERSE_FLAG);
	pcert->hash_algorithm = get_oid_info(CRYPT_OID_INFO_OID_KEY, pcrypt_cert->pCertInfo->SignatureAlgorithm.pszObjId, 0);
	pcert->algorithm = get_oid_info(CRYPT_OID_INFO_OID_KEY, pcrypt_cert->pCertInfo->SubjectPublicKeyInfo.Algorithm.pszObjId, 0);
	
	std::vector<BYTE, crypt_allocator> buffer;
	if (0 == get_cert_property(pcrypt_cert, CERT_HASH_PROP_ID, buffer)) {
		CRYPT_INTEGER_BLOB blob = { static_cast<DWORD>(buffer.size()), buffer.data() };
		pcert->thumbprint = format_crypt_blob(::blob);
	}
	if (0 == get_cert_property(pcrypt_cert, CERT_SIGNATURE_HASH_PROP_ID, buffer)) {
		CRYPT_INTEGER_BLOB blob = { static_cast<DWORD>(buffer.size()), buffer.data() };
		pcert->signature_hash = format_crypt_blob(&blob);
	}
	if (0 == get_cert_property(pcrypt_cert, CERT_SUBJECT_PUB_KEY_BIT_LENGTH_PROP_ID, buffer)) {
		pcert->publickey_length = *reinterpret_cast<DWORD*>(buffer.data());
	}
	if (0 == get_cert_property(pcrypt_cert, CERT_CTL_USAGE_PROP_ID, buffer)) {
		std::vector<BYTE, crypt_allocator> cert_usage;
		if (0 == crypt_decode(X509_ENHANCED_KEY_USAGE, buffer, cert_usage)) {
			decode_usage(reinterpret_cast<CTL_USAGE*>(cert_usage.data()), pcert->usages);
		}
	}

	DWORD size = 0;
	if (CertGetEnhancedKeyUsage(pcrypt_cert, 0, 0, &size)) {
		buffer.resize(size);
		if (CertGetEnhancedKeyUsage(pcrypt_cert, 0, reinterpret_cast<PCERT_ENHKEY_USAGE>(buffer.data()), &size)) {
			decode_usage(reinterpret_cast<CERT_ENHKEY_USAGE*>(buffer.data()), pcert->enhanced_key_usages);
		}
	}

	pcert->publickey = format_crypt_blob(reinterpret_cast<CRYPT_INTEGER_BLOB*>(::pcrypt_cert->pCertInfo->SubjectPublicKeyInfo.PublicKey));
	
	return 0;
}

void oid_publisher(CRYPT_ATTRIBUTE& crypt_attr, ATTR& attr, SIGNATURE* psignature) {
	DWORD size = 0;
	if (!CryptDecodeObject(X509_ASN_ENCODING | PKCS_7_ASN_ENCODING, SPC_SP_OPUS_INFO_OBJID,
		crypt_attr.rgValue[0].pbData, crypt_attr.rgValue[0].cbData, 0, 0, &size))
		return;
	std::vector<BYTE, crypt_allocator> opusinfo(size);
	if (!CryptDecodeObject(X509_ASN_ENCODING | PKCS_7_ASN_ENCODING, SPC_SP_OPUS_INFO_OBJID,
		crypt_attr.rgValue[0].pbData, crypt_attr.rgValue[0].cbData, 0, opusinfo.data(), &size))
		return;

	const PSPC_SP_OPUS_INFO popus = reinterpret_cast<PSPC_SP_OPUS_INFO>(opusinfo.data());
	if (popus->pwszProgramName) {
		attr.value += L"program:";
		attr.value += popus->pwszProgramName;
		attr.value += L"; ";
	}
	if (popus->pPublisherInfo) {
		switch (popus->pPublisherInfo->dwLinkChoice) {
		case SPC_URL_LINK_CHOICE: 
			attr.value += L"publisher:";
			attr.value += popus->pPublisherInfo->pwszUrl;
			attr.value += L"; ";
			break;
		case SPC_FILE_LINK_CHOICE: 
			attr.value += L"publisher:";
			attr.value += popus->pPublisherInfo->pwszFile;
			attr.value += L"; ";
			break;
		}
	}
	if (popus->pMoreInfo) {
		switch (popus->pMoreInfo->dwLinkChoice) {
		case SPC_URL_LINK_CHOICE: 
			attr.value += L"moreinfo:";
			attr.value += popus->pMoreInfo->pwszUrl;
			attr.value += L"; ";
			break;
		case SPC_FILE_LINK_CHOICE: 
			attr.value += L"moreinfo:";
			attr.value += popus->pMoreInfo->pwszFile;
			attr.value += L"; ";
			break;
		}
	}
}

void oid_signingtime(CRYPT_ATTRIBUTE& crypt_attr, ATTR& attr, SIGNATURE* psignature) {
	FILETIME ft = {};
	DWORD size = sizeof(ft);
	if (CryptDecodeObject(X509_ASN_ENCODING | PKCS_7_ASN_ENCODING, szOID_RSA_signingTime, 
						  crypt_attr.rgValue[0].pbData, crypt_attr.rgValue[0].cbData, 0, &ft, &size)) {
		attr.value += L"timestamp:";
		attr.value += string_from_filetime(::ft);
		attr.value += L"; ";
	}
}

void oid_general(CRYPT_ATTRIBUTE& crypt_attr, ATTR& attr, SIGNATURE* psignature) {
	attr.value = format_crypt_blob(::crypt_attr.rgValue[0]);
}

std::map<std::string, void(*)(CRYPT_ATTRIBUTE::, ATTR::, SIGNATURE*)> oidfuncs = {
	{ SPC_SP_OPUS_INFO_OBJID, oid_publisher },
	{ szOID_RSA_signingTime, oid_signingtime }
};

void process_attrs(CRYPT_ATTRIBUTES& crypt_attrs, attrs_t& attrs, SIGNATURE* psignature) {
	for (DWORD index = 0; index < crypt_attrs.cAttr; ++index) {
		std::unique_ptr<ATTR> attr(std::make_unique<ATTR>());
		std::wstring_convert<std::codecvt_utf8_utf16<wchar_t>> conv;
		attr->oid = conv.from_bytes(crypt_attrs.rgAttr[index].pszObjId);
		attr->size = crypt_attrs.rgAttr[index].rgValue->cbData;
		
		PCCRYPT_OID_INFO pcrypt_oidinfo = CryptFindOIDInfo(CRYPT_OID_INFO_OID_KEY, crypt_attrs.rgAttr[index].pszObjId, 0);
		if (pcrypt_oidinfo && pcrypt_oidinfo->pwszName)
			attr->name = pcrypt_oidinfo->pwszName;
		else
			attr->name = attr->oid;

		auto& it = oidfuncs.find(crypt_attrs.rgAttr[index].pszObjId);
		if (it != oidfuncs.end())
			it->second(crypt_attrs.rgAttr[index], *attr.get(), psignature);
		else
			oid_general(crypt_attrs.rgAttr[index], *attr.get(), psignature);

		attrs.insert(std::make_pair(attr->oid, std::move(attr)));
	}
}

DWORD get_signature_from_CMSG_SIGNER_INFO(CMSG_SIGNER_INFO* pcrypt_signature, HCERTSTORE hcrypt_certstore, SIGNATURE* psignature) {
	psignature->issuer = get_cert_name(::pcrypt_signature->Issuer, CERT_X500_NAME_STR | CERT_NAME_STR_REVERSE_FLAG);
	psignature->version = pcrypt_signature->dwVersion;
	psignature->serial_number = format_crypt_blob(::pcrypt_signature->SerialNumber, true);
	psignature->encryption_algorithm = get_oid_info(CRYPT_OID_INFO_OID_KEY, pcrypt_signature->HashEncryptionAlgorithm.pszObjId, 0);
	psignature->hash_algorithm = get_oid_info(CRYPT_OID_INFO_OID_KEY, pcrypt_signature->HashAlgorithm.pszObjId, 0);

	CERT_INFO certinfo = {};
	certinfo.Issuer = pcrypt_signature->Issuer;
	certinfo.SerialNumber = pcrypt_signature->SerialNumber;
	CERT_CONTEXT_t cert_ctx(CertFindCertificateInStore(hcrypt_certstore, X509_ASN_ENCODING | PKCS_7_ASN_ENCODING, 0, CERT_FIND_SUBJECT_CERT, &certinfo, 0));
	if (!cert_ctx)
		return GetLastError();

	std::unique_ptr<CERT> cert(std::make_unique<CERT>());
	DWORD last_error = get_cert_from_CERT_CONTEXT(cert_ctx.get(), cert.get());
	if (last_error != 0)
		return last_error;

	CERT_CHAIN_PARA crypt_chain_para = { sizeof(CERT_CHAIN_PARA) };
	crypt_chain_para.RequestedUsage.dwType = USAGE_MATCH_TYPE_AND;
	CERT_CHAIN_CONTEXT_t cert_chain_ctx;
	if (!CertGetCertificateChain(0, cert_ctx.get(), 0, 0, &crypt_chain_para, 0, 0, get_pointer_wrapper(cert_chain_ctx)))
		return GetLastError();

	if (cert_chain_ctx->cChain > 0 && cert_chain_ctx->rgpChain[0]->cElement > 1) {
		CERT* pcert_current = cert.get();
		for (DWORD index = 1; index < cert_chain_ctx->rgpChain[0]->cElement; ++index) {
			pcert_current->chained_cert = std::make_unique<CERT>();
			if ((last_error = get_cert_from_CERT_CONTEXT(cert_chain_ctx->rgpChain[0]->rgpElement[index]->pCertContext, pcert_current->chained_cert.get())) != 0)
				return last_error;
			pcert_current = pcert_current->chained_cert.get();
		}
	}

	process_attrs(pcrypt_signature->AuthAttrs, psignature->attrs_auth, psignature);
	process_attrs(pcrypt_signature->UnauthAttrs, psignature->attrs_unauth, psignature);

	psignature->cert = std::move(cert);

	return 0;
}

DWORD get_signature_from_wtstatedata(HANDLE state_data, SIGNATURE* psignature) {
	CRYPT_PROVIDER_DATA* pcrypt_prov_data = WTHelperProvDataFromStateData(state_data);

	std::unique_ptr<SIGNATURE> countersign(std::make_unique<SIGNATURE>());
	BOOL is_countersign[2] = { FALSE, TRUE };
	SIGNATURE* psignatures[2] = { psignature, countersign.get() };
	for (size_t n = 0; n < _countof(is_countersign); ++n) {
		CRYPT_PROVIDER_SGNR *pcrypt_signature = WTHelperGetProvSignerFromChain(pcrypt_prov_data, 0, is_countersign[n], 0);
		if (!pcrypt_signature)
			continue;
		CRYPT_PROVIDER_CERT *pcrypt_cert = WTHelperGetProvCertFromChain(pcrypt_signature, 0);
		if (!pcrypt_cert)
			continue;

		HCERTSTORE_t cert_store(CertOpenStore(CERT_STORE_PROV_COLLECTION, X509_ASN_ENCODING | PKCS_7_ASN_ENCODING, 0, 0, 0));
		if (!cert_store)
			return GetLastError();

		for (DWORD index = 0; index < pcrypt_prov_data->chStores; ++index) {
			if (!CertAddStoreToCollection(cert_store.get(), pcrypt_prov_data->pahStores[index], 0, 0))
				return GetLastError();
		}

		DWORD last_error = get_signature_from_CMSG_SIGNER_INFO(pcrypt_signature->psSigner, cert_store.get(), psignatures[n]);
		if (last_error != 0)
			return last_error;
	}
	psignature->countersign = std::move(countersign);
	return 0;
}

unsigned int get_file_signatures(SIGNATURE_INFO& siginfo) {
	WINTRUST_DATA wt_data = { sizeof(WINTRUST_DATA) };
	wt_data.dwUIChoice = WTD_UI_NONE;
	wt_data.fdwRevocationChecks = WTD_REVOKE_NONE;
	wt_data.dwUnionChoice = WTD_CHOICE_FILE;

	WINTRUST_FILE_INFO wt_fileinfo = { sizeof(WINTRUST_FILE_INFO) };
	wt_fileinfo.pcwszFilePath = siginfo.target.c_str();
	wt_data.pFile = &wt_fileinfo;

	WINTRUST_SIGNATURE_SETTINGS wt_sig_settings = { sizeof(WINTRUST_SIGNATURE_SETTINGS) };
	wt_sig_settings.dwFlags = WSS_GET_SECONDARY_SIG_COUNT | WSS_VERIFY_SPECIFIC;
	wt_data.pSignatureSettings = &wt_sig_settings;

	GUID wt_guid = WINTRUST_ACTION_GENERIC_VERIFY_V2;
	DWORD last_error = 0;
	while (true) {
		wt_data.dwStateAction = WTD_STATEACTION_VERIFY;
		if ((last_error = WinVerifyTrust(NULL, &wt_guid, &wt_data)) != 0)
			return last_error;

		std::unique_ptr<SIGNATURE> signature = std::make_unique<SIGNATURE>();
		if ((last_error = get_signature_from_wtstatedata(wt_data.hWVTStateData, signature.get())) != 0)
			return last_error;
		siginfo.signatures.push_back(std::move(signature));

		wt_data.dwStateAction = WTD_STATEACTION_CLOSE;
		if ((last_error = WinVerifyTrust(NULL, &wt_guid, &wt_data)) != 0)
			return last_error;
		wt_data.hWVTStateData = NULL;

		if (++wt_data.pSignatureSettings->dwIndex > wt_data.pSignatureSettings->cSecondarySigs)
			break;
	}
	return 0;
}

unsigned int get_file_signatures(const wchar_t* file, SIGNATURE_INFO& catalog, SIGNATURE_INFO& embedded) {
	embedded.target = file;
	embedded.result = get_file_signatures(embedded);

	HCATADMIN_t catadmin;
	const GUID guid = DRIVER_ACTION_VERIFY;
	if (!CryptCATAdminAcquireContext(get_pointer_wrapper(catadmin), &guid, 0))
		return (catalog.result = GetLastError());
	HANDLE_t h(CreateFile(file, GENERIC_READ, FILE_SHARE_READ, 0, OPEN_EXISTING, 0, 0));
	if (!h)
		return (catalog.result = GetLastError());

	DWORD size = 0;
	if (!CryptCATAdminCalcHashFromFileHandle(h.get(), &size, 0, 0))
		return (catalog.result = GetLastError());
	std::vector<BYTE, crypt_allocator> buffer(size);
	if(!CryptCATAdminCalcHashFromFileHandle(h.get(), &size, buffer.data(), 0))
		return (catalog.result = GetLastError());

	HCATINFO_t hcatinfo(CryptCATAdminEnumCatalogFromHash(catadmin.get(), buffer.data(), static_cast<DWORD>(buffer.size()), 0, 0), deleter_HCATINFO(catadmin.get()));
	if(!hcatinfo)
		return (catalog.result = GetLastError());

	CATALOG_INFO catinfo = {};
	if (!CryptCATCatalogInfoFromContext(hcatinfo.get(), &catinfo, 0))
		return (catalog.result = GetLastError());

	catalog.target = catinfo.wszCatalogFile;
	catalog.result = get_file_signatures(catalog);

	return 0;
}

signature.h

#pragma once

#include <string>
#include <list>
#include <map>
#include <vector>
#include <memory>

using pairs = std::vector<std::pair<std::wstring, std::wstring>>;

struct CERT {
	unsigned int version;
	std::wstring serial_number;
	std::wstring name;
	std::wstring issuer;
	std::wstring subject;
	std::wstring algorithm;
	std::wstring hash_algorithm;
	std::wstring thumbprint;
	std::wstring signature_hash;
	std::wstring publickey;
	unsigned long publickey_length;
	pairs usages;
	pairs enhanced_key_usages;
	std::wstring valid_from;
	std::wstring valid_to;

	std::unique_ptr<CERT> chained_cert;
};

struct ATTR {
	unsigned int size;
	std::wstring oid;
	std::wstring name;
	std::wstring value;
};
using attrs_t = std::map<std::wstring, std::unique_ptr<ATTR>>;

struct SIGNATURE {
	unsigned int version;
	std::wstring issuer;
	std::wstring serial_number;
	std::wstring encryption_algorithm;
	std::wstring hash_algorithm;
	attrs_t attrs_auth;
	attrs_t attrs_unauth;
	std::unique_ptr<CERT> cert;
	std::unique_ptr<SIGNATURE> countersign;
};

using signatures_t = std::list<std::unique_ptr<SIGNATURE>>;

struct SIGNATURE_INFO {
	std::wstring target;
	signatures_t signatures;
	unsigned int result;
};

unsigned int get_file_signatures(const wchar_t* file, SIGNATURE_INFO& catalog, SIGNATURE_INFO& embedded);

test.cpp

#include "signature.h"

#include <string>
#include <list>
#include <vector>
#include <iterator>
#include <stdarg.h>

using strings_t = std::vector<std::wstring>;

std::wstring formatstring(const wchar_t* format, ...) {
	va_list arguments;
	va_start(arguments, format);
	int size = _vscwprintf(format, arguments) + 1;
	std::unique_ptr<wchar_t[]> buffer(new wchar_t[size]);
	vswprintf_s(buffer.get(), size, format, arguments);
	va_end(arguments);
	return std::wstring(buffer.get());
}

strings_t format_cert(const CERT* pcert) {
	strings_t strings;
	strings.push_back(formatstring(L"version: %lu", pcert->version));
	strings.push_back(formatstring(L"serial number: %s", pcert->serial_number.c_str()));
	strings.push_back(formatstring(L"name: %s", pcert->name.c_str()));
	strings.push_back(formatstring(L"issuer: %s", pcert->issuer.c_str()));
	strings.push_back(formatstring(L"subject: %s", pcert->subject.c_str()));
	strings.push_back(formatstring(L"algorithm: %s", pcert->algorithm.c_str()));
	strings.push_back(formatstring(L"hashing: %s", pcert->hash_algorithm.c_str()));
	strings.push_back(formatstring(L"thumbprint: %s", pcert->thumbprint.c_str()));
	strings.push_back(formatstring(L"signature hash: %s", pcert->signature_hash.c_str()));
	strings.push_back(formatstring(L"public key (%lu bits): %s", pcert->publickey_length, pcert->publickey.c_str()));
	strings.push_back(formatstring(L"valid from: %s", pcert->valid_from.c_str()));
	strings.push_back(formatstring(L"valid to: %s", pcert->valid_to.c_str()));
	if (!pcert->usages.empty()) {
		strings.push_back(formatstring(L"%zu cert usage(s) ->", pcert->usages.size()));
		for (const auto& it : pcert->usages) {
			strings.push_back(formatstring(L"%zs : %zs", it.first.c_str(), it.second.c_str()));
		}
	}
	if (!pcert->enhanced_key_usages.empty()) {
		strings.push_back(formatstring(L"%zu extended key usage(s) ->", pcert->enhanced_key_usages.size()));
		for (const auto& it : pcert->enhanced_key_usages) {
			strings.push_back(formatstring(L"%zs : %zs", it.first.c_str(), it.second.c_str()));
		}
	}
	return strings;
}

strings_t format_signature(const SIGNATURE* psignature) {
	strings_t strings;
	strings.push_back(formatstring(L"version: %lu", psignature->version));
	strings.push_back(formatstring(L"issuer: %s", psignature->issuer.c_str()));
	strings.push_back(formatstring(L"serial number: %s", psignature->serial_number.c_str()));
	strings.push_back(formatstring(L"algorithm: %s", psignature->encryption_algorithm.c_str()));
	strings.push_back(formatstring(L"hashing: %s", psignature->hash_algorithm.c_str()));
	strings.push_back(L"authenticated attributes ->");
	for (const auto& it : psignature->attrs_auth) {
		const std::wstring name = it.second->oid == it.second->name ? it.second->oid : formatstring(L"%s(%s)", it.second->oid.c_str(), it.second->name.c_str());
		strings.push_back(formatstring(L"%s(%lu bytes): %s", name.c_str(), it.second->size, it.second->value.c_str()));
	}
	strings.push_back(L"unauthenticated attributes ->");
	for (const auto& it : psignature->attrs_unauth) {
		const std::wstring name = it.second->oid == it.second->name ? it.second->oid : formatstring(L"%s(%s)", it.second->oid.c_str(), it.second->name.c_str());
		strings.push_back(formatstring(L"%s(%lu bytes): %s", name.c_str(), it.second->size, it.second->value.c_str()));
	}
	return strings;
}

void print_with_indent(const std::wstring& indent, const strings_t& strings) {
	for (const auto& it : strings) {
		wprintf(L"%s%s\n", indent.c_str(), it.c_str());
	}
}

void print_certs(std::wstring indent, const CERT* pcert) {
	while (true) {
		indent += L'\t';
		print_with_indent(indent, format_cert(pcert));
		pcert = pcert->chained_cert.get();
		if (pcert == nullptr)
			break;

		wprintf(L"\n");
		print_with_indent(indent, { L"chained certificate ->" });
	}
}

void print_signature(std::wstring indent, const SIGNATURE* psignature) {
	print_with_indent(indent, format_signature(psignature));
	if (psignature->cert) {
		wprintf(L"\n");
		print_with_indent(indent, { L"signing certificate ->" });
		indent += L'\t';
		print_certs(indent, psignature->cert.get());
	}
}

void print_signatures(std::wstring indent, const signatures_t& signatures) {
	size_t index = 0;
	for (auto& it : signatures) {
		wprintf(L"\n");
		print_with_indent(indent, { formatstring(L"signature %zu of %zu ->", ++index, signatures.size()) });
		indent += L'\t';
		print_signature(indent, it.get());

		if (it->countersign) {
			wprintf(L"\n");
			print_with_indent(indent, { L"counter-signing signature ->" });
			wprintf(L"\n");
			indent += L'\t';
			print_signature(indent, it->countersign.get());
		}
	}
}

int wmain(int argc, wchar_t* argv[])
{
	if (argc < 2) {
		wprintf(L"error: no file specified\n");
		return -1;
	}
	SIGNATURE_INFO catalog, embedded;
	get_file_signatures(argv[1], catalog, embedded);
	if (catalog.result == 0 && !catalog.signatures.empty()) {
		wprintf(L"%s is catalog signed by %s ->", argv[1], catalog.target.c_str());
		wprintf(L"\n\n");
		print_signatures(L"\t", catalog.signatures);
		wprintf(L"\n");
	}
	if (embedded.result == 0 && !embedded.signatures.empty()) {
		wprintf(L"%s has %zu embedded signature(s) ->", embedded.target.c_str(), embedded.signatures.size());
		wprintf(L"\n\n");
		print_signatures(L"\t", embedded.signatures);
		wprintf(L"\n");
	}    
	return 0;
}

Test output for C:\Windows\system32\msvcp140.dll:

C:\Windows\system32\msvcp140.dll has 2 embedded signature(s) ->


	signature 1 of 2 ->
		version: 1
		issuer: CN=Microsoft Code Signing PCA, O=Microsoft Corporation, L=Redmond, S=Washington, C=US
		serial number: 330000010a2c79aed7797ba6ac00010000010a
		algorithm: RSA
		hashing: sha1
		authenticated attributes ->
		1.2.840.113549.1.9.3(Content Type)(12 bytes): 060a2b060104018237020104
		1.2.840.113549.1.9.4(Message Digest)(22 bytes): 04147c9005d13c561c514a5491bc6d23cd381c623e24
		1.3.6.1.4.1.311.2.1.11(14 bytes): 300c060a2b060104018237020115
		1.3.6.1.4.1.311.2.1.12(128 bytes): program:msvcp140_c7c6faa1-8830-48f3-a151-4445245658c0.dll; moreinfo:http://microsoft.com; 
		unauthenticated attributes ->
		1.2.840.113549.1.9.6(Counter Sign)(533 bytes): 3082021102010130818e3077310b30090603550406130255533113301106035504...
		1.3.6.1.4.1.311.2.4.1(9287 bytes): 3082244306092a864886f70d010702a082243430822430020101310f300d060960...

		signing certificate ->
				version: 2
				serial number: 330000010a2c79aed7797ba6ac00010000010a
				name: Microsoft Corporation
				issuer: Microsoft Code Signing PCA
				subject: CN=Microsoft Corporation, OU=MOPR, O=Microsoft Corporation, L=Redmond, S=Washington, C=US
				algorithm: RSA
				hashing: sha1RSA
				thumbprint: 3bda323e552db1fde5f4fbee75d6d5b2b187eedc
				public key (2048 bits): 3082010a028201010092fc6f36bbe103b928d646d1a261baa546fb63163e2ef947...
				valid from: Sun Jun  4 13:42:45 2015
				valid to: Sun Sep  4 13:42:45 2016
				1 extended key usage(s) ->
				Code Signing : 1.3.6.1.5.5.7.3.3

				chained certificate ->
					version: 2
					serial number: 6133261a000000000031
					name: Microsoft Code Signing PCA
					issuer: Microsoft Root Certificate Authority
					subject: CN=Microsoft Code Signing PCA, O=Microsoft Corporation, L=Redmond, S=Washington, C=US
					algorithm: RSA
					hashing: sha1RSA
					thumbprint: 3caf9ba2db5570caf76942ff99101b993888e257
					public key (2048 bits): 3082010a0282010100b272595c193064bf1d9a602020429976536c3e1bd66fcccb...
					valid from: Sun Aug 31 18:19:32 2010
					valid to: Sun Aug 31 18:29:32 2020

					chained certificate ->
						version: 2
						serial number: 79ad16a14aa0a5ad4c7358f407132e65
						name: Microsoft Root Certificate Authority
						issuer: Microsoft Root Certificate Authority
						subject: CN=Microsoft Root Certificate Authority, DC=microsoft, DC=com
						algorithm: RSA
						hashing: sha1RSA
						thumbprint: cdd4eeae6000ac7f40c3802c171e30148030c072
						public key (4096 bits): 3082020a0282020100f35dfa8067d45aa7a90c2c9020d035083c7584cdb707899c...
						valid from: Sun May  9 19:19:22 2001
						valid to: Sun May  9 19:28:13 2021

		counter-signing signature ->

			version: 1
			issuer: CN=Microsoft Time-Stamp PCA, O=Microsoft Corporation, L=Redmond, S=Washington, C=US
			serial number: 330000009ceefe1455a95d355000000000009c
			algorithm: sha1RSA
			hashing: sha1
			authenticated attributes ->
			1.2.840.113549.1.9.3(Content Type)(11 bytes): 06092a864886f70d010701
			1.2.840.113549.1.9.4(Message Digest)(22 bytes): 0414e5d07b4a560a1b8f6fb4a6e9b5646a9274a11243
			1.2.840.113549.1.9.5(Signing Time)(15 bytes): timestamp:Sun Jul 13 15:35:02 2016; 
			unauthenticated attributes ->

			signing certificate ->
					version: 2
					serial number: 330000009ceefe1455a95d355000000000009c
					name: Microsoft Time-Stamp Service
					issuer: Microsoft Time-Stamp PCA
					subject: CN=Microsoft Time-Stamp Service, OU=nCipher DSE ESN:5847-F761-4F70, OU=MOPR, O=Microsoft Corporation, L=Redmond, S=Washington, C=US
					algorithm: RSA
					hashing: sha1RSA
					thumbprint: ea1ed0f0ea5bd51d62e2306bb430917e750fa6b9
					public key (2048 bits): 3082010a0282010100cc2586c97e887a048ffbe4954f5ea532c069beb1ad4d4659...
					valid from: Sun Mar 30 15:21:30 2016
					valid to: Sun Jun 30 15:21:30 2017
					1 extended key usage(s) ->
					Time Stamping : 1.3.6.1.5.5.7.3.8

					chained certificate ->
						version: 2
						serial number: 6116683400000000001c
						name: Microsoft Time-Stamp PCA
						issuer: Microsoft Root Certificate Authority
						subject: CN=Microsoft Time-Stamp PCA, O=Microsoft Corporation, L=Redmond, S=Washington, C=US
						algorithm: RSA
						hashing: sha1RSA
						thumbprint: 375fcb825c3dc3752a02e34eb70993b4997191ef
						public key (2048 bits): 3082010a02820101009fa16cb1dfdb48922a7c6b2e19e1bde2e3c599512350adce...
						valid from: Sun Apr  3 08:53:09 2007
						valid to: Sun Apr  3 09:03:09 2021
						1 extended key usage(s) ->
						Time Stamping : 1.3.6.1.5.5.7.3.8

						chained certificate ->
							version: 2
							serial number: 79ad16a14aa0a5ad4c7358f407132e65
							name: Microsoft Root Certificate Authority
							issuer: Microsoft Root Certificate Authority
							subject: CN=Microsoft Root Certificate Authority, DC=microsoft, DC=com
							algorithm: RSA
							hashing: sha1RSA
							thumbprint: cdd4eeae6000ac7f40c3802c171e30148030c072
							public key (4096 bits): 3082020a0282020100f35dfa8067d45aa7a90c2c9020d035083c7584cdb707899c...
							valid from: Sun May  9 19:19:22 2001
							valid to: Sun May  9 19:28:13 2021

			signature 2 of 2 ->
				version: 1
				issuer: CN=Microsoft Code Signing PCA 2011, O=Microsoft Corporation, L=Redmond, S=Washington, C=US
				serial number: 330000006447849486db411938000000000064
				algorithm: RSA
				hashing: sha256
				authenticated attributes ->
				1.2.840.113549.1.9.25.4(3 bytes): 020101
				1.2.840.113549.1.9.3(Content Type)(12 bytes): 060a2b060104018237020104
				1.2.840.113549.1.9.4(Message Digest)(34 bytes): 0420924b8ccfb6c85cd67b5486e4975b6531908db6340217e6dc309456681e0c4e...
				1.3.6.1.4.1.311.2.1.11(14 bytes): 300c060a2b060104018237020115
				1.3.6.1.4.1.311.2.1.12(128 bytes): program:msvcp140_c7c6faa1-8830-48f3-a151-4445245658c0.dll; moreinfo:http://microsoft.com; 
				unauthenticated attributes ->
				1.3.6.1.4.1.311.3.3.1(4920 bytes): 3082133406092a864886f70d010702a082132530821321020103310f300d060960...

				signing certificate ->
						version: 2
						serial number: 330000006447849486db411938000000000064
						name: Microsoft Corporation
						issuer: Microsoft Code Signing PCA 2011
						subject: CN=Microsoft Corporation, OU=MOPR, O=Microsoft Corporation, L=Redmond, S=Washington, C=US
						algorithm: RSA
						hashing: sha256RSA
						thumbprint: 54da79490495aa6d7898b183d86ea600e3fc5ebf
						public key (2048 bits): 3082010a0282010100932edad8e63e98d91b0f645f4da8584d143bf77a84fc3c11...
						valid from: Sun Oct 28 16:31:46 2015
						valid to: Sun Jan 28 16:31:46 2017
						2 extended key usage(s) ->
						Code Signing : 1.3.6.1.5.5.7.3.3
						Microsoft Publisher : 1.3.6.1.4.1.311.76.8.1

						chained certificate ->
							version: 2
							serial number: 610e90d2000000000003
							name: Microsoft Code Signing PCA 2011
							issuer: Microsoft Root Certificate Authority 2011
							subject: CN=Microsoft Code Signing PCA 2011, O=Microsoft Corporation, L=Redmond, S=Washington, C=US
							algorithm: RSA
							hashing: sha256RSA
							thumbprint: f252e794fe438e35ace6e53762c0a234a2c52135
							public key (4096 bits): 3082020a0282020100abf0fa72101c2eadd86eaa82104d34baf2b658219f421b2a...
							valid from: Sun Jul  8 16:59:09 2011
							valid to: Sun Jul  8 17:09:09 2026

							chained certificate ->
								version: 2
								serial number: 3f8bc8b5fc9fb29643b569d66c42e144
								name: Microsoft Root Certificate Authority 2011
								issuer: Microsoft Root Certificate Authority 2011
								subject: CN=Microsoft Root Certificate Authority 2011, O=Microsoft Corporation, L=Redmond, S=Washington, C=US
								algorithm: RSA
								hashing: sha256RSA
								thumbprint: 8f43288ad272f3103b6fb1428485ea3014c0bcfe
								public key (4096 bits): 3082020a0282020100b28041aa35384d13723268224db8b2f1ffd552bc6cc7f5d2...
								valid from: Sun Mar 22 18:05:28 2011
								valid to: Sun Mar 22 18:13:04 2036

				counter-signing signature ->

					version: 1
					issuer: CN=Microsoft Time-Stamp PCA 2010, O=Microsoft Corporation, L=Redmond, S=Washington, C=US
					serial number: 3300000084938a428f2c7c23e8000000000084
					algorithm: sha256RSA
					hashing: sha256
					authenticated attributes ->
					1.2.840.113549.1.9.16.2.12(210 bytes): 3081cf3081cc3081b10414f72b6e73ace8b7384ad64657646a49150bcc367d3081...
					1.2.840.113549.1.9.3(Content Type)(13 bytes): 060b2a864886f70d0109100104
					1.2.840.113549.1.9.4(Message Digest)(34 bytes): 042033e9a68c8d31df8c28f10ef373cb8ead2057f1e86705d226bb6acf34051e27...
					unauthenticated attributes ->

					signing certificate ->
							version: 2
							serial number: 3300000084938a428f2c7c23e8000000000084
							name: Microsoft Time-Stamp Service
							issuer: Microsoft Time-Stamp PCA 2010
							subject: CN=Microsoft Time-Stamp Service, OU=nCipher DSE ESN:7AFA-E41C-E142, OU=MOPR, O=Microsoft Corporation, L=Redmond, S=Washington, C=US
							algorithm: RSA
							hashing: sha256RSA
							thumbprint: f72b6e73ace8b7384ad64657646a49150bcc367d
							public key (2048 bits): 3082010a0282010100cba6479458fc671e863671f72dae98997f45d2346775e2ea...
							valid from: Sun Mar 30 15:24:23 2016
							valid to: Sun Jun 30 15:24:23 2017
							1 extended key usage(s) ->
							Time Stamping : 1.3.6.1.5.5.7.3.8

							chained certificate ->
								version: 2
								serial number: 6109812a000000000002
								name: Microsoft Time-Stamp PCA 2010
								issuer: Microsoft Root Certificate Authority 2010
								subject: CN=Microsoft Time-Stamp PCA 2010, O=Microsoft Corporation, L=Redmond, S=Washington, C=US
								algorithm: RSA
								hashing: sha256RSA
								thumbprint: 2aa752fe64c49abe82913c463529cf10ff2f04ee
								public key (2048 bits): 3082010a0282010100a91d0dbc77118a3a20ecfc1397f5fa7f69946b745410d5a5...
								valid from: Sun Jul  1 17:36:55 2010
								valid to: Sun Jul  1 17:46:55 2025

								chained certificate ->
									version: 2
									serial number: 28cc3a25bfba44ac449a9b586b4339aa
									name: Microsoft Root Certificate Authority 2010
									issuer: Microsoft Root Certificate Authority 2010
									subject: CN=Microsoft Root Certificate Authority 2010, O=Microsoft Corporation, L=Redmond, S=Washington, C=US
									algorithm: RSA
									hashing: sha256RSA
									thumbprint: 3b1efd3a66ea28b16697394703a72ca340a05bd5
									public key (4096 bits): 3082020a0282020100b9089e28e4e4ec064e5068b341c57bebaeb68eaf81ba2244...
									valid from: Sun Jun 23 17:57:24 2010
									valid to: Sun Jun 23 18:04:01 2035


Test output for C:\Windows\system32\drivers\netio.sys:

C:\Windows\system32\drivers\netio.sys is catalog signed by C:\WINDOWS\system32\CatRoot\{F750E6C3-38EE-11D1-85E5-00C04FC295EE}\Microsoft-Windows-Foundation-Group-minio-Package~31bf3856ad364e35~amd64~~10.0.14393.0.cat ->


	signature 1 of 1 ->
		version: 1
		issuer: CN=Microsoft Windows Production PCA 2011, O=Microsoft Corporation, L=Redmond, S=Washington, C=US
		serial number: 33000000bce120fdd27cc8ee930000000000bc
		algorithm: RSA
		hashing: sha256
		authenticated attributes ->
		1.2.840.113549.1.9.3(Content Type)(11 bytes): 06092b0601040182370a01
		1.2.840.113549.1.9.4(Message Digest)(34 bytes): 042063c4f438e5d341bf48cea142e34ff3ca1007d5528f6346fb0870c9197ed741...
		1.3.6.1.4.1.311.2.1.11(14 bytes): 300c060a2b060104018237020115
		1.3.6.1.4.1.311.2.1.12(76 bytes): program:Microsoft Windows; moreinfo:http://www.microsoft.com/windows; 
		unauthenticated attributes ->
		1.3.6.1.4.1.311.3.3.1(4921 bytes): 3082133506092a864886f70d010702a082132630821322020103310f300d060960...

		signing certificate ->
				version: 2
				serial number: 33000000bce120fdd27cc8ee930000000000bc
				name: Microsoft Windows
				issuer: Microsoft Windows Production PCA 2011
				subject: CN=Microsoft Windows, O=Microsoft Corporation, L=Redmond, S=Washington, C=US
				algorithm: RSA
				hashing: sha256RSA
				thumbprint: e85459b23c232db3cb94c7a56d47678f58e8e51e
				public key (2048 bits): 3082010a02820101009a901e3ef75f68c07e45c7885c08fb5c41e083bc40aef9c5...
				valid from: Sun Aug 18 13:15:28 2015
				valid to: Sun Nov 18 13:15:28 2016
				2 extended key usage(s) ->
				Code Signing : 1.3.6.1.5.5.7.3.3
				Windows System Component Verification : 1.3.6.1.4.1.311.10.3.6

				chained certificate ->
					version: 2
					serial number: 61077656000000000008
					name: Microsoft Windows Production PCA 2011
					issuer: Microsoft Root Certificate Authority 2010
					subject: CN=Microsoft Windows Production PCA 2011, O=Microsoft Corporation, L=Redmond, S=Washington, C=US
					algorithm: RSA
					hashing: sha256RSA
					thumbprint: 580a6f4cc4e4b669b9ebdc1b2b3e087b80d0678d
					public key (2048 bits): 3082010a0282010100dd0cbba2e42e09e3e7c5f79669bc0021bd693333efad04cb...
					valid from: Sun Oct 19 14:41:42 2011
					valid to: Sun Oct 19 14:51:42 2026

					chained certificate ->
						version: 2
						serial number: 28cc3a25bfba44ac449a9b586b4339aa
						name: Microsoft Root Certificate Authority 2010
						issuer: Microsoft Root Certificate Authority 2010
						subject: CN=Microsoft Root Certificate Authority 2010, O=Microsoft Corporation, L=Redmond, S=Washington, C=US
						algorithm: RSA
						hashing: sha256RSA
						thumbprint: 3b1efd3a66ea28b16697394703a72ca340a05bd5
						public key (4096 bits): 3082020a0282020100b9089e28e4e4ec064e5068b341c57bebaeb68eaf81ba2244...
						valid from: Sun Jun 23 17:57:24 2010
						valid to: Sun Jun 23 18:04:01 2035

		counter-signing signature ->

			version: 1
			issuer: CN=Microsoft Time-Stamp PCA 2010, O=Microsoft Corporation, L=Redmond, S=Washington, C=US
			serial number: 3300000096bff2b421c0777524000000000096
			algorithm: sha256RSA
			hashing: sha256
			authenticated attributes ->
			1.2.840.113549.1.9.16.2.12(210 bytes): 3081cf3081cc3081b10414e46ac0933041367b85960b0b5c65c81d8bc0bd9e3081...
			1.2.840.113549.1.9.3(Content Type)(13 bytes): 060b2a864886f70d0109100104
			1.2.840.113549.1.9.4(Message Digest)(34 bytes): 04209d0651812070f975444042420665b97b3355625432ae310a358b61596786a0...
			unauthenticated attributes ->

			signing certificate ->
					version: 2
					serial number: 3300000096bff2b421c0777524000000000096
					name: Microsoft Time-Stamp Service
					issuer: Microsoft Time-Stamp PCA 2010
					subject: CN=Microsoft Time-Stamp Service, OU=nCipher DSE ESN:C0F4-3086-DEF8, OU=MOPR, O=Microsoft Corporation, L=Redmond, S=Washington, C=US
					algorithm: RSA
					hashing: sha256RSA
					thumbprint: e46ac0933041367b85960b0b5c65c81d8bc0bd9e
					public key (2048 bits): 3082010a02820101009d3230b4bc8b2645a2ebecb0908986ee22231d20bac45f29...
					valid from: Sun Apr 27 13:06:17 2016
					valid to: Sun Jul 27 13:06:17 2017
					1 extended key usage(s) ->
					Time Stamping : 1.3.6.1.5.5.7.3.8

					chained certificate ->
						version: 2
						serial number: 6109812a000000000002
						name: Microsoft Time-Stamp PCA 2010
						issuer: Microsoft Root Certificate Authority 2010
						subject: CN=Microsoft Time-Stamp PCA 2010, O=Microsoft Corporation, L=Redmond, S=Washington, C=US
						algorithm: RSA
						hashing: sha256RSA
						thumbprint: 2aa752fe64c49abe82913c463529cf10ff2f04ee
						public key (2048 bits): 3082010a0282010100a91d0dbc77118a3a20ecfc1397f5fa7f69946b745410d5a5...
						valid from: Sun Jul  1 17:36:55 2010
						valid to: Sun Jul  1 17:46:55 2025

						chained certificate ->
							version: 2
							serial number: 28cc3a25bfba44ac449a9b586b4339aa
							name: Microsoft Root Certificate Authority 2010
							issuer: Microsoft Root Certificate Authority 2010
							subject: CN=Microsoft Root Certificate Authority 2010, O=Microsoft Corporation, L=Redmond, S=Washington, C=US
							algorithm: RSA
							hashing: sha256RSA
							thumbprint: 3b1efd3a66ea28b16697394703a72ca340a05bd5
							public key (4096 bits): 3082020a0282020100b9089e28e4e4ec064e5068b341c57bebaeb68eaf81ba2244...
							valid from: Sun Jun 23 17:57:24 2010
							valid to: Sun Jun 23 18:04:01 2035

C:\Windows\system32\drivers\netio.sys has 1 embedded signature(s) ->


	signature 1 of 1 ->
		version: 1
		issuer: CN=Microsoft Windows Production PCA 2011, O=Microsoft Corporation, L=Redmond, S=Washington, C=US
		serial number: 33000000bce120fdd27cc8ee930000000000bc
		algorithm: RSA
		hashing: sha256
		authenticated attributes ->
		1.2.840.113549.1.9.3(Content Type)(12 bytes): 060a2b060104018237020104
		1.2.840.113549.1.9.4(Message Digest)(34 bytes): 042045e2044dd404430791e7561120136c0f6992aeb8e51304dc5ed49c90db692a...
		1.3.6.1.4.1.311.2.1.11(14 bytes): 300c060a2b060104018237020115
		1.3.6.1.4.1.311.2.1.12(76 bytes): program:Microsoft Windows; moreinfo:http://www.microsoft.com/windows; 
		unauthenticated attributes ->
		1.3.6.1.4.1.311.3.3.1(4921 bytes): 3082133506092a864886f70d010702a082132630821322020103310f300d060960...

		signing certificate ->
				version: 2
				serial number: 33000000bce120fdd27cc8ee930000000000bc
				name: Microsoft Windows
				issuer: Microsoft Windows Production PCA 2011
				subject: CN=Microsoft Windows, O=Microsoft Corporation, L=Redmond, S=Washington, C=US
				algorithm: RSA
				hashing: sha256RSA
				thumbprint: e85459b23c232db3cb94c7a56d47678f58e8e51e
				public key (2048 bits): 3082010a02820101009a901e3ef75f68c07e45c7885c08fb5c41e083bc40aef9c5...
				valid from: Sun Aug 18 13:15:28 2015
				valid to: Sun Nov 18 13:15:28 2016
				2 extended key usage(s) ->
				Code Signing : 1.3.6.1.5.5.7.3.3
				Windows System Component Verification : 1.3.6.1.4.1.311.10.3.6

				chained certificate ->
					version: 2
					serial number: 61077656000000000008
					name: Microsoft Windows Production PCA 2011
					issuer: Microsoft Root Certificate Authority 2010
					subject: CN=Microsoft Windows Production PCA 2011, O=Microsoft Corporation, L=Redmond, S=Washington, C=US
					algorithm: RSA
					hashing: sha256RSA
					thumbprint: 580a6f4cc4e4b669b9ebdc1b2b3e087b80d0678d
					public key (2048 bits): 3082010a0282010100dd0cbba2e42e09e3e7c5f79669bc0021bd693333efad04cb...
					valid from: Sun Oct 19 14:41:42 2011
					valid to: Sun Oct 19 14:51:42 2026

					chained certificate ->
						version: 2
						serial number: 28cc3a25bfba44ac449a9b586b4339aa
						name: Microsoft Root Certificate Authority 2010
						issuer: Microsoft Root Certificate Authority 2010
						subject: CN=Microsoft Root Certificate Authority 2010, O=Microsoft Corporation, L=Redmond, S=Washington, C=US
						algorithm: RSA
						hashing: sha256RSA
						thumbprint: 3b1efd3a66ea28b16697394703a72ca340a05bd5
						public key (4096 bits): 3082020a0282020100b9089e28e4e4ec064e5068b341c57bebaeb68eaf81ba2244...
						valid from: Sun Jun 23 17:57:24 2010
						valid to: Sun Jun 23 18:04:01 2035

		counter-signing signature ->

			version: 1
			issuer: CN=Microsoft Time-Stamp PCA 2010, O=Microsoft Corporation, L=Redmond, S=Washington, C=US
			serial number: 3300000084938a428f2c7c23e8000000000084
			algorithm: sha256RSA
			hashing: sha256
			authenticated attributes ->
			1.2.840.113549.1.9.16.2.12(210 bytes): 3081cf3081cc3081b10414f72b6e73ace8b7384ad64657646a49150bcc367d3081...
			1.2.840.113549.1.9.3(Content Type)(13 bytes): 060b2a864886f70d0109100104
			1.2.840.113549.1.9.4(Message Digest)(34 bytes): 04209b2522fc952ec9be3a0dfb592356b999baacb2d7dabcd9d25d52f4dbed84db...
			unauthenticated attributes ->

			signing certificate ->
					version: 2
					serial number: 3300000084938a428f2c7c23e8000000000084
					name: Microsoft Time-Stamp Service
					issuer: Microsoft Time-Stamp PCA 2010
					subject: CN=Microsoft Time-Stamp Service, OU=nCipher DSE ESN:7AFA-E41C-E142, OU=MOPR, O=Microsoft Corporation, L=Redmond, S=Washington, C=US
					algorithm: RSA
					hashing: sha256RSA
					thumbprint: f72b6e73ace8b7384ad64657646a49150bcc367d
					public key (2048 bits): 3082010a0282010100cba6479458fc671e863671f72dae98997f45d2346775e2ea...
					valid from: Sun Mar 30 15:24:23 2016
					valid to: Sun Jun 30 15:24:23 2017
					1 extended key usage(s) ->
					Time Stamping : 1.3.6.1.5.5.7.3.8

					chained certificate ->
						version: 2
						serial number: 6109812a000000000002
						name: Microsoft Time-Stamp PCA 2010
						issuer: Microsoft Root Certificate Authority 2010
						subject: CN=Microsoft Time-Stamp PCA 2010, O=Microsoft Corporation, L=Redmond, S=Washington, C=US
						algorithm: RSA
						hashing: sha256RSA
						thumbprint: 2aa752fe64c49abe82913c463529cf10ff2f04ee
						public key (2048 bits): 3082010a0282010100a91d0dbc77118a3a20ecfc1397f5fa7f69946b745410d5a5...
						valid from: Sun Jul  1 17:36:55 2010
						valid to: Sun Jul  1 17:46:55 2025

						chained certificate ->
							version: 2
							serial number: 28cc3a25bfba44ac449a9b586b4339aa
							name: Microsoft Root Certificate Authority 2010
							issuer: Microsoft Root Certificate Authority 2010
							subject: CN=Microsoft Root Certificate Authority 2010, O=Microsoft Corporation, L=Redmond, S=Washington, C=US
							algorithm: RSA
							hashing: sha256RSA
							thumbprint: 3b1efd3a66ea28b16697394703a72ca340a05bd5
							public key (4096 bits): 3082020a0282020100b9089e28e4e4ec064e5068b341c57bebaeb68eaf81ba2244...
							valid from: Sun Jun 23 17:57:24 2010
							valid to: Sun Jun 23 18:04:01 2035

Error from Process Monitor: “Unable to write PROCMON23.SYS”

Today I was trying to start a boot logging with Process Monitor on Windows 10 and received following error:

Unable to write PROCMON23.SYS. Make sure that you have permission to write to the %%SystemRoot%%\System32\Drivers directory.

Unable to write PROCMON23.SYS

Sharing violation was observed when removed procmon.exe and procmon64.exe from filters:
Sharing Violation

The workaround is go into C:\Windows\System32\drivers and rename procmon23.sys to something else, then try to enable boot logging from options menu again, it would succeed this time.

Get process command line

There is one officially supported way of getting command line from other processes, that is through WMI Win32_Process class. Here is a command that will display command line and pid of running processes:

WMIC PROCESS get Commandline,Processid

It is also possible to get command line of other processes by reading their memory and track down the information through path PEB -> ProcessParameters -> CommandLine. It involves using unsupported API or structures, and on 64-bit machines extra care needs to be taken for pointer size and structure alignment. Here is the code that can be built and run from 32-bit or 64-bit processes:

#include "stdafx.h"
#include <stddef.h>
#include <Windows.h>
#include <Winternl.h>

typedef NTSTATUS (NTAPI *PFN_NtQueryInformationProcess)(
	HANDLE ProcessHandle,
	DWORD ProcessInformationClass,
	PVOID ProcessInformation,
	DWORD ProcessInformationLength,
	PDWORD ReturnLength
	);

#ifdef _WIN64
	typedef struct _UNICODE_STRING_32 {
			USHORT Length;
			USHORT MaximumLength;
			UINT Buffer;
		} UNICODE_STRING_32;
	typedef struct _RTL_USER_PROCESS_PARAMETERS_32 {
		BYTE Reserved1[16];
		UINT32 Reserved2[10];
		UNICODE_STRING_32 ImagePathName;
		UNICODE_STRING_32 CommandLine;
	} RTL_USER_PROCESS_PARAMETERS_32;
#else
	typedef struct _UNICODE_STRING_64 {
			USHORT Length;
			USHORT MaximumLength;
			UINT32 pad;
			UINT64 Buffer;
		} UNICODE_STRING_64;
	typedef struct _PROCESS_BASIC_INFORMATION_64 {
		UINT64 Reserved1;
		UINT64 PebBaseAddress;
		UINT64 Reserved2[2];
		UINT64 UniqueProcessId;
		UINT64 Reserved3;
	} PROCESS_BASIC_INFORMATION_64;
	typedef struct _RTL_USER_PROCESS_PARAMETERS_64 {
		BYTE Reserved1[16];
		UINT64 Reserved2[10];
		UNICODE_STRING_64 ImagePathName;
		UNICODE_STRING_64 CommandLine;
	} RTL_USER_PROCESS_PARAMETERS_64;

	typedef NTSTATUS(NTAPI *PFN_NtWow64ReadVirtualMemory64)(
		HANDLE ProcessHandle,
		UINT64 BaseAddress,
		PVOID Buffer,
		ULONG64 Size,
		PULONG64 NumberOfBytesRead);

	PFN_NtQueryInformationProcess g_pfnNtQueryInformationProcess64 = NULL;
	PFN_NtWow64ReadVirtualMemory64 g_pfnWow64ReadVirtualMemory64 = NULL;
#endif

PFN_NtQueryInformationProcess g_pfnNtQueryInformationProcess = NULL;
SYSTEM_INFO g_sysinfo = { 0 };

void get_process_info_initialize()
{
	g_pfnNtQueryInformationProcess = (PFN_NtQueryInformationProcess)GetProcAddress(GetModuleHandle(L"ntdll.dll"), "NtQueryInformationProcess");
#ifndef _WIN64
	g_pfnNtQueryInformationProcess64 = (PFN_NtQueryInformationProcess)GetProcAddress(GetModuleHandle(L"ntdll.dll"), "NtWow64QueryInformationProcess64");
	g_pfnWow64ReadVirtualMemory64 = (PFN_NtWow64ReadVirtualMemory64)GetProcAddress(GetModuleHandle(L"ntdll.dll"), "NtWow64ReadVirtualMemory64");
#endif

	GetNativeSystemInfo(&g_sysinfo);
}

LONG get_process_info_by_ntqueryinformationprocess(HANDLE hprocess, PWSTR *info, DWORD pi_class, DWORD info_max_size)
{
	LONG error = 0;
	PCHAR buffer = 0;
	do
	{
		DWORD info_size = 0;
		error = g_pfnNtQueryInformationProcess(hprocess, pi_class, 0, 0, &info_size);
		if (info_size == 0 || info_size > info_max_size)
			break;

		buffer = (PCHAR)malloc(info_size);
		if (buffer == 0)
		{
			error = ERROR_NOT_ENOUGH_MEMORY;
			break;
		}

		error = g_pfnNtQueryInformationProcess(hprocess, pi_class, buffer, info_size, &info_size);
		if(!NT_SUCCESS(error))
			break;

		PUNICODE_STRING str = (PUNICODE_STRING)buffer;
		size_t size = str->Length + sizeof(WCHAR);
		PWSTR temp_str = (PWSTR)malloc(size);
		if (temp_str == 0)
		{
			error = ERROR_NOT_ENOUGH_MEMORY;
			break;
		}
		memset(temp_str, 0, size);
		memcpy(temp_str, str->Buffer, str->Length);
		*info = temp_str;

		error = 0;
	}while(0);
	if (buffer != 0)
		free(buffer);
	return error;
}

template <class unicode_string_type>
LONG read_process_memory_unicode_string(HANDLE hprocess, LPCVOID remote_address, PWSTR *output)
{
	LONG error = 0;
	do
	{
		unicode_string_type str = { 0 };
		ReadProcessMemory(hprocess, remote_address, &str, sizeof(str), 0);
		error = GetLastError();
		if (str.Length == 0 || str.Buffer == 0)
			break;

		size_t size = str.Length + sizeof(WCHAR);
		PWSTR temp_str = (PWSTR)malloc(size);
		if (temp_str == 0)
		{
			error = ERROR_NOT_ENOUGH_MEMORY;
			break;
		}

		memset(temp_str, 0, size);
		ReadProcessMemory(hprocess, (PCHAR)str.Buffer, temp_str, str.Length, 0);
		*output = temp_str;

		error = 0;
	} while (0);
	return error;
}

LONG get_process_info_by_peb_native(HANDLE hprocess, PWSTR *image, PWSTR *command_line)
{
	LONG error = 0;
	do
	{
		PROCESS_BASIC_INFORMATION pbi = {0};
		error = g_pfnNtQueryInformationProcess(hprocess, ProcessBasicInformation, &pbi, sizeof(pbi), NULL);
		if(pbi.PebBaseAddress == 0)
			break;

		SIZE_T process_parameter = 0;
		ReadProcessMemory(hprocess, (PCHAR)pbi.PebBaseAddress + offsetof(PEB, ProcessParameters), &process_parameter, sizeof(process_parameter), 0);
		error = GetLastError();
		if (process_parameter == 0)
			break;

		if (image != 0)
			error = read_process_memory_unicode_string<UNICODE_STRING>(hprocess, (PCHAR)process_parameter + offsetof(RTL_USER_PROCESS_PARAMETERS, ImagePathName), image);
		
		if (command_line != 0)
			error = read_process_memory_unicode_string<UNICODE_STRING>(hprocess, (PCHAR)process_parameter + offsetof(RTL_USER_PROCESS_PARAMETERS, CommandLine), command_line);
	}while(0);
	return error;
}

#ifdef _WIN64
#if 1
LONG get_process_info_by_peb_to_wow64(HANDLE hprocess, PWSTR *image, PWSTR *command_line)
{
	LONG error = 0;
	do
	{
		UINT64 peb32 = 0;
		error = g_pfnNtQueryInformationProcess(hprocess, 26 /*ProcessWow64Information*/, &peb32, sizeof(peb32), NULL);
		if(peb32 == 0)
			break;

		UINT64 process_parameter = 0;
		ReadProcessMemory(hprocess, (PCHAR)peb32 + 0x10, &process_parameter, sizeof(process_parameter), 0);
		error = GetLastError();
		if (process_parameter == 0)
			break;

		if (image != 0)
			error = read_process_memory_unicode_string<UNICODE_STRING_32>(hprocess, (PCHAR)process_parameter + offsetof(RTL_USER_PROCESS_PARAMETERS_32, ImagePathName), image);

		if (command_line != 0)
			error = read_process_memory_unicode_string<UNICODE_STRING_32>(hprocess, (PCHAR)process_parameter + offsetof(RTL_USER_PROCESS_PARAMETERS_32, CommandLine), command_line);
	}while(0);
	return error;
}
#endif
#endif

#ifndef _WIN64
template <typename unicode_string_type>
LONG read_virtual_memory_from_wow64_unicode_string(HANDLE hprocess, UINT64 remote_address, PWSTR *output)
{
	LONG error = 0;
	do
	{
		unicode_string_type str = { 0 };
		g_pfnWow64ReadVirtualMemory64(hprocess, remote_address, &str, sizeof(str), 0);
		error = GetLastError();
		if (str.Length == 0 || str.Buffer == 0)
			break;

		size_t size = str.Length + sizeof(WCHAR);
		PWSTR temp_str = (PWSTR)malloc(size);
		if (temp_str == 0)
		{
			error = ERROR_NOT_ENOUGH_MEMORY;
			break;
		}

		memset(temp_str, 0, size);
		g_pfnWow64ReadVirtualMemory64(hprocess, str.Buffer, temp_str, str.Length, 0);
		*output = temp_str;

		error = 0;
	} while (0);
	return error;
}

LONG get_process_info_by_peb_from_wow64(HANDLE hprocess, PWSTR *image, PWSTR *command_line)
{
	LONG error = 0;
	do
	{
		PROCESS_BASIC_INFORMATION_64 pbi = {0};
		error = g_pfnNtQueryInformationProcess64(hprocess, ProcessBasicInformation, &pbi, sizeof(pbi), NULL);
		if(pbi.PebBaseAddress == 0)
			break;

		UINT64 process_parameter = 0;
		g_pfnWow64ReadVirtualMemory64(hprocess, (UINT64)(pbi.PebBaseAddress + 0x20), &process_parameter, sizeof(process_parameter), 0);
		error = GetLastError();
		if(process_parameter == 0)
			break;

		if (image != 0)
			error = read_virtual_memory_from_wow64_unicode_string<UNICODE_STRING_64>(hprocess, process_parameter + offsetof(RTL_USER_PROCESS_PARAMETERS_64, ImagePathName), image);
		
		if(command_line != 0)
			error = read_virtual_memory_from_wow64_unicode_string<UNICODE_STRING_64>(hprocess, process_parameter + offsetof(RTL_USER_PROCESS_PARAMETERS_64, CommandLine), command_line);
	}while(0);
	return error;
}
#endif

LONG get_process_commandline(DWORD pid, BOOL *is32bit, LPWSTR *image, LPWSTR *command_line)
{
	LONG error = 0;
	HANDLE hprocess = NULL;
	do
	{
		DWORD access[] = { PROCESS_QUERY_INFORMATION | PROCESS_QUERY_LIMITED_INFORMATION | PROCESS_VM_READ,
			PROCESS_QUERY_INFORMATION | PROCESS_QUERY_LIMITED_INFORMATION,
			PROCESS_QUERY_LIMITED_INFORMATION };

		for (size_t n = 0; n < _countof(access); ++n)
		{
			hprocess = OpenProcess(access[n], FALSE, pid);
			error = GetLastError();
			if (hprocess != NULL)
				break;
		}
		if (NULL == hprocess)
			break;

#ifdef _WIN64
		*is32bit = FALSE;
#else
		*is32bit = TRUE;
#endif
		IsWow64Process(hprocess, is32bit);

#ifdef _WIN64
		{
			if (*is32bit)
				error = get_process_info_by_peb_to_wow64(hprocess, image, command_line);
			else
				error = get_process_info_by_peb_native(hprocess, image, command_line);
		}
#else
		{
			if (*is32bit || g_sysinfo.wProcessorArchitecture == PROCESSOR_ARCHITECTURE_INTEL)
				error = get_process_info_by_peb_native(hprocess, image, command_line);
			else
				error = get_process_info_by_peb_from_wow64(hprocess, image, command_line);
		}
#endif

		if (image == 0)
			error = get_process_info_by_ntqueryinformationprocess(hprocess, image, 43 /* ProcessImageFileNameWin32 */, 2048);

		if (command_line == 0)
			error = get_process_info_by_ntqueryinformationprocess(hprocess, command_line, 60 /* ProcessCommandLineInformation */, 2048);
	} while (0);
	if (hprocess != NULL)
		CloseHandle(hprocess);
	return error;
}

Here is the testing code:

// cmdline.cpp : Defines the entry point for the console application.
//

#include "stdafx.h"
#include <iostream>
#include <memory>
#include <map>
#include <stack>
#include <algorithm>

#include <Windows.h>
#include <Psapi.h>
#include <tlhelp32.h>
#pragma comment (lib, "Psapi.lib")

void get_process_info_initialize();
LONG get_process_commandline(DWORD pid, BOOL *is32bit, LPWSTR *image, LPWSTR *command_line);

struct _PROCESS_INFO;
typedef std::map<DWORD, _PROCESS_INFO*> processlist;
typedef std::stack<_PROCESS_INFO*> processtack;

typedef struct _PROCESS_INFO
{
	DWORD pid;
	BOOL is32bit;
	PWSTR image;
	PWSTR command_line;
	processlist child_processes;
}PROCESS_INFO, *PPROCESS_INFO;

PPROCESS_INFO process_alloc();
void process_free(PPROCESS_INFO process);

PPROCESS_INFO process_alloc()
{
	PPROCESS_INFO process = new(std::nothrow)PROCESS_INFO;
	if (process != 0)
	{
		process->pid = 0;
		process->is32bit = 0;
		process->image = 0;
		process->command_line = 0;
	}
	return process;
}

void process_free(PPROCESS_INFO process)
{
	processtack st;
	st.push(process);
	while (!st.empty())
	{
		process = st.top();
		st.pop();
		std::for_each(process->child_processes.begin(), process->child_processes.end(), [&st](processlist::value_type& val){ st.push(val.second); });

		if (process->image != 0)
			free(process->image);
		if (process->command_line != 0)
			free(process->command_line);
		delete process;
	}
}

#ifndef _WIN64
BOOL is_running_as_wow64()
{
	BOOL is_wow64 = FALSE;
	typedef BOOL(WINAPI *PFN_IsWow64Process) (HANDLE, PBOOL);
	PFN_IsWow64Process pfnIsWow64Process = (PFN_IsWow64Process)GetProcAddress(GetModuleHandle(L"kernel32"), "IsWow64Process");
	if (pfnIsWow64Process != NULL)
		pfnIsWow64Process(GetCurrentProcess(), &is_wow64);
	return is_wow64;
}
#endif

PPROCESS_INFO get_process_list()
{
	PPROCESS_INFO root = process_alloc();
	do
	{
		if (root == 0)
			break;

		HANDLE hProcessSnap = CreateToolhelp32Snapshot(TH32CS_SNAPPROCESS, 0);
		if (INVALID_HANDLE_VALUE == hProcessSnap)
			break;

		processlist processes;

		PROCESSENTRY32 pe32 = { 0 };
		pe32.dwSize = sizeof(PROCESSENTRY32);
		Process32First(hProcessSnap, &pe32);
		do
		{
			if (pe32.th32ProcessID == 0)
				continue;

			PPROCESS_INFO process = process_alloc();
			if (process == NULL)
				break;

			process->pid = pe32.th32ProcessID;
			PWSTR path = pe32.szExeFile;

			MODULEENTRY32 me32 = { 0 };
			me32.dwSize = sizeof(MODULEENTRY32);
			HANDLE hModuleSnap = CreateToolhelp32Snapshot(TH32CS_SNAPMODULE, pe32.th32ProcessID);
			if (INVALID_HANDLE_VALUE != hModuleSnap && Module32First(hModuleSnap, &me32))
				path = me32.szExePath;

			get_process_commandline(process->pid, &process->is32bit, &process->image, &process->command_line);

			if (process->image == 0)
				process->image = _wcsdup(path);

			if (INVALID_HANDLE_VALUE != hModuleSnap)
				CloseHandle(hModuleSnap);

			processlist::iterator itor = processes.find(pe32.th32ParentProcessID);
			if (itor == processes.end())
			{
				root->child_processes.insert(std::make_pair(pe32.th32ProcessID, process));
			}
			else
			{
				itor->second->child_processes.insert(std::make_pair(pe32.th32ProcessID, process));
			}
			processes.insert(std::make_pair(pe32.th32ProcessID, process));
		} while (Process32Next(hProcessSnap, &pe32));

		CloseHandle(hProcessSnap);
	} while (0);
	return root;
}

void dump_process_list(const processlist& processes)
{
	std::wstring indent;
	processtack st;
	std::for_each(processes.rbegin(), processes.rend(), [&st](const processlist::value_type& val){ st.push(val.second); });
	while (!st.empty())
	{
		const PPROCESS_INFO process = st.top();
		st.pop();
		if (process == 0)
		{
			indent.resize(indent.size() - 1);
			continue;
		}
		wprintf(indent.c_str());
		wprintf(L"pid:%d image=%s, cmdline=%s\n\n", process->pid, process->image, process->command_line);

		indent += L'\t';
		st.push(0);
		std::for_each(process->child_processes.rbegin(), process->child_processes.rend(), [&st](const processlist::value_type& val){ st.push(val.second); });
	}
}

void enable_debug_privilege(LPCWSTR priviledge)
{
	HANDLE token = 0;
	OpenProcessToken(GetCurrentProcess(), TOKEN_ADJUST_PRIVILEGES | TOKEN_QUERY, &token);
	if (token != 0)
	{
		TOKEN_PRIVILEGES privileges = { 0 };
		privileges.PrivilegeCount = 1;
		privileges.Privileges[0].Attributes = SE_PRIVILEGE_ENABLED;
		LookupPrivilegeValue(0, priviledge, &privileges.Privileges[0].Luid);
		AdjustTokenPrivileges(token, FALSE, &privileges, 0, 0, 0);
		CloseHandle(token);
	}
}

int _tmain(int argc, _TCHAR* argv[])
{
	enable_debug_privilege(SE_DEBUG_NAME);
	get_process_info_initialize();
	PPROCESS_INFO root = get_process_list();
	if (root != 0)
	{
		dump_process_list(root->child_processes);
		process_free(root);
	}
	return 0;
}

Looking for the terminated thread leaving loader lock orphaned

Sometimes a process hang is caused by doing something dangerous while holding the loader lock, if you can get a dump file, it is pretty easy to run !locks command to find out the locks causing the trouble. It can be a little bit complicated if one of the locking mechanism is not a critical section(for example, event or mutex), but it is still manageable by looking for similar function calls among the stacks. However today I ran into a dump file from a hung process which shows the loader lock was locked by a thread that has been terminated and being left orphaned:

0:000> !locks
CritSec ntdll!LdrpLoaderLock+0 at 00007ffa0aa3d8a8
WaiterWoken        No
LockCount          9
RecursionCount     1
OwningThread       710
EntryCount         0
ContentionCount    39
*** Locked

Scanned 76 critical sections

0:000> ~~[710]k
             ^ Illegal thread error in '~~[710]k'

Most likely, thread 710 was terminated while it was holding the loader lock. Because of this, there were many other threads in same process were being blocked and waiting for it to be unlocked, which unfortunately will never happen. Since the thread 710 no longer exists, there was no TEB, we don’t know what it was running when being terminated, and most importantly, we don’t know who terminated it. If you have worked on certain amount of crash dumps, eventually you will run into some internal NT structures, one of which is _CLIENT_ID:

0:000> dt _CLIENT_ID             
combase!_CLIENT_ID
   +0x000 UniqueProcess    : Ptr64 Void
   +0x008 UniqueThread     : Ptr64 Void

It is simply a structure of process id and thread id, and it is being used by many other structures like _TEB. It has been a technique to search them in memory in order to identify the parent structure that contains them. Let’s try this here to see what we can get.

0:000> |
.  0	id: 608	examine	name: C:\Program Files\ABC\xyz.exe

0:000> s 0 L?ffffffffffffffff 00 00 00 00 08 06 00 00 00 00 00 00 10 07 00 00
00007ffa`07eb6214  00 00 00 00 08 06 00 00-00 00 00 00 10 07 00 00  ................

First we use “|” command to get process id 608, then use the “s” command to search for memory that can be interpreted as a _CLIENT_ID structure that has 608 as process id and 710 as thread id. “s” command did return an address that contains matching data, let’s find out what it is:

0:000> !address 00007ffa`07eb6214
...
Usage:                  Image
Base Address:           00007ffa`07eb3000
End Address:            00007ffa`07eb7000
Region Size:            00000000`00004000
State:                  00001000	MEM_COMMIT
Protect:                00000004	PAGE_READWRITE
Type:                   01000000	MEM_IMAGE
Allocation Base:        00007ffa`07dc0000
Allocation Protect:     00000080	PAGE_EXECUTE_WRITECOPY
Image Path:             C:\Windows\System32\KERNELBASE.dll
Module Name:            KERNELBASE
Loaded Image Name:      KERNELBASE.dll
Mapped Image Name:      
More info:              lmv m KERNELBASE
More info:              !lmi KERNELBASE
More info:              ln 0x7ffa07eb6214
More info:              !dh 0x7ffa07dc0000

0:000> ln 0x7ffa07eb6214
(00007ffa`07eb6200)   KERNELBASE!BaseTerminatedLoaderLockOwner+0x14   |  (00007ffa`07eb6268)   KERNELBASE!BaseDataFileHandleTableElementCount

It is very surprising, the memory found belongs to a global data structure of KERNELBASE.dll, and based on the name given by symbol, looks like Windows is tracking the threads if they were terminated when holding the loader lock. Could it save the stack as well?

0:000> dps 00007ffa`07eb6200
00007ffa`07eb6200  00000000`00000001
00007ffa`07eb6208  00000000`00000608
00007ffa`07eb6210  00000000`00000c64
00007ffa`07eb6218  00000000`00000608
00007ffa`07eb6220  00000000`00000710
00007ffa`07eb6228  00007ffa`07e64263 KERNELBASE!TerminateThread+0xaf
00007ffa`07eb6230  00007ff9`fa494995 somedll+0x4995
00007ffa`07eb6238  00007ff9`fa492421 somedll+0x2421
00007ffa`07eb6240  00007ff9`fa4924e9 somedll+0x24e9
00007ffa`07eb6248  00007ffa`0a1113d2 kernel32!BaseThreadInitThunk+0x22
00007ffa`07eb6250  00007ffa`0a9803c4 ntdll!RtlUserThreadStart+0x34
00007ffa`07eb6258  00000000`00000000
00007ffa`07eb6260  00000000`00000000
00007ffa`07eb6268  00000000`00000000
00007ffa`07eb6270  00000000`00000000
00007ffa`07eb6278  00000000`00000000

It did! After fixing symbols for somedll and re-run dps command, I received the full stack of the thread when it is calling TerminateThread when the thread being terminated was holding loader lock, it’s time to notify the team responsible for somedll.

Bonus read: Raymond has another two excellent articles about finding ghost threads causing crashing from kernel or user dumps:

How to view the stack of threads that were terminated as part of process teardown from the kernel debugger

How to view the stack of threads that were terminated as part of process teardown from user mode

Crash from HeapLock

One of my routine work is investigating crash dumps collected by Windows Error Reporting service (also known as Winqual). This morning I found cabs are available for some of high volume buckets, I downloaded the first one, opened it in WinDbg, corrected symbols, and here is the stack after .ecxr and kb command:

0:019> kb
  *** Stack trace for last set context - .thread/.cxr resets it
ChildEBP RetAddr  Args to Child              
01a1ac80 74b76dee 01c70000 01a1ed04 62d54ab3 ntdll!RtlLockHeap+0x16
01a1ac8c 62d54ab3 01c70000 bfb3fcb0 01a1f050 KERNELBASE!HeapLock+0xe
01a1ed04 62d55239 01b42b71 00000200 01a1f064 MyModule!IsAddressOnHeap+0x83
...

Here is the pseudo code for MyModule!IsAddressOnHeap, it takes a parameter of void*, then walk through heaps in current process and returns whether the address might be coming from a heap.

BOOL IsAddressOnHeap(void* p) {
	HANDLE handles[4096];
	int heaps = GetProcessHeaps(sizeof(handles)/sizeof(HANDLE), handles);	
	if(heaps == 0)
		return FALSE;
	for(int i=0 ; i < heaps ; i++) {
		if (HeapLock(handles[i])) {
			__try {
				// use HeapWalk to determine whether the address is on heap
			}
			__finally {
				HeapUnlock(handles[i]);
			}
		}
	}
	return FALSE;
}

I already had a good guess about what went wrong, but let’s confirm it. First, let’s see how the execution was transferred from MyModule!IsAddressOnHeap to KERNELBASE!HeapLock:

0:019> ub 62d54ab3 
MyModule!IsAddressOnHeap+0x62
62d54a92 0f8447010000    je      MyModule!IsAddressOnHeap+0x1af (62d54bdf)
62d54a98 33db            xor     ebx,ebx
62d54a9a 895de4          mov     dword ptr [ebp-1Ch],ebx
62d54a9d 3bdf            cmp     ebx,edi
62d54a9f 0f8d34010000    jge     MyModule!IsAddressOnHeap+0x1a9 (62d54bd9)
62d54aa5 8b8c9da4bfffff  mov     ecx,dword ptr [ebp+ebx*4-405Ch]
62d54aac 51              push    ecx
62d54aad ff155ca0d662    call    dword ptr [MyModule!_imp__HeapLock (62d6a05c)]

The parameter passed to KERNELBASE!HeapLock was pushed to stack, which came from ecx, which in turn came from address at ebp+ebx*4-405Ch. Also we know that ebx is used as the index (variable i) and edi is used as the counter (variable heaps). We can get the value of registers from context record saved at the time of crash, but they might not be the same as they were in MyModule!IsAddressOnHeap considering there are two more functions (KERNELBASE!HeapLock+0xe and ntdll!RtlLockHeap+0x16) on the stack above MyModule!IsAddressOnHeap. Things gets simple in this case as these two functions didn’t run much (offsets are 0xe and 0x16), which means we can disassemble them from start to the offset to make sure ebx and edi were not changed (I did it and they were not changed). In case they were changed and since they are non-volatile registers, caller is responsible in preserving the old values (by pushing/pop them to/from stack) so they will still be retrievable. Since they are not changed in this case, let’s use the registers in crash context record to see what were in handles array:

0:019> .ecxr
eax=3b94b284 ebx=00000004 ecx=01c70000 edx=00000000 esi=01c70000 edi=00000005
eip=77481641 esp=01a1ac54 ebp=01a1ac80 iopl=0         nv up ei pl nz na pe nc
cs=001b  ss=0023  ds=0023  es=0023  fs=003b  gs=0000             efl=00010206
ntdll!RtlLockHeap+0x16:
77481641 f7464400000001  test    dword ptr [esi+44h],1000000h ds:0023:01c70044=????????

* 01a1ed04 is ChildEBP of MyModule!IsAddressOnHeap from kb command, the output is in sync with first arg from kb
0:019> dd 01a1ed04+4*4-405c l1
01a1acb8  01c70000

* edi has value of 5, dump first 5 elements in array heaps
0:019> dd 01a1ed04-405c l5
01a1aca8  006d0000 004b0000 00690000 01a80000
01a1acb8  01c70000

Now we know what were in array heaps. Let’s take a look at the heaps in PEB:

0:019> dt _PEB @$peb
MyModule!_PEB
   ...
   +0x088 NumberOfHeaps    : 4
   +0x08c MaximumNumberOfHeaps : 0x10
   +0x090 ProcessHeaps     : 0x7754d3a0  -> 0x006d0000 Void
   ...
0:019> dd 0x7754d3a0 l4
7754d3a0  006d0000 004b0000 00690000 01a80000

So there were 4 heaps reported from PEB, while our array had 5, the first 4 were same as in PEB, the fifth one was not in PEB heaps and wasn’t in a committed page:

0:019> !address 01c70000
Usage:                  Free
Base Address:           01b48000
End Address:            5ffd0000
Region Size:            5e488000
State:                  00010000	MEM_FREE
Protect:                00000001	PAGE_NOACCESS
Type:                   <info not present at the target>

So it looks like by the time function MyModule!IsAddressOnHeap walked to the last heap, it had been destroyed and memory had been freed. We need to put HeapLock into another SEH block to catch the access violation, existing block cannot be reused as it sole purpose is to use the __finally block to unlock the heap (if it was locked successfully).

I wrote a simple test application and validated the theory: after calling HeapDestroy on a heap, calling HeapLock with same heap handle will cause access violation. HeapValidate can be used to check if the heap was valid (even with a handle that has been used with HeapDestroy), I though of calling it before HeapLock, but ditched the idea quickly due to the potential racing condition that HeapDestroy could called in-between.

Get this Pointer from 64-bit crash dump

64-bit calling convention means that under most scenarios, the first four parameters are passed using rcx, rdx, r8 and r9 registers rather than using stack. It’s difficult to find out actual parameters passed to functions in the calling stack from a 64-bit crash dump. CodeMachine has a great article X64 Deep Dive on this topic, it also talks about x64 compiler optimization, exception handling, parameter passing as well as parameter retrieval. It is a must read if crash dump analysis is part of your job.

There are lots of times we want to find out “this” pointer from the crash dump. “this” pointer is passed using rcx, and if the function accesses class members, very likely rcx will be saved into rsi or rdi after function’s prolog. I created a MFC application using VS2012 project wizard, inserted DebugBreak() at the end of CMFCApplication1App::InitInstance, built it using x64|Release configuration, ran it, save the crash dump and open it with WinDbg, this is what the initial stack looks like:

0:000> kL
Child-SP          RetAddr           Call Site
000000cc`aa65e468 000007fb`01df12d2 ntdll!NtWaitForMultipleObjects+0xa
000000cc`aa65e470 000007fb`0211d20e KERNELBASE!WaitForMultipleObjectsEx+0xe5
000000cc`aa65e750 000007fb`0211cfd2 kernel32!WerpReportFaultInternal+0x1fa
000000cc`aa65e7f0 000007fb`01e6fc87 kernel32!WerpReportFault+0x42
000000cc`aa65e820 000007fb`04db9183 KERNELBASE!UnhandledExceptionFilter+0x1d7
000000cc`aa65e920 000007fb`04ce4fea ntdll! ?? ::FNODOBFM::`string'+0x9e3
000000cc`aa65e950 000007fb`04ce464d ntdll!_C_specific_handler+0x8e
000000cc`aa65e9c0 000007fb`04ce567c ntdll!RtlpExecuteHandlerForException+0xd
000000cc`aa65e9f0 000007fb`04cc4bba ntdll!RtlDispatchException+0x392
000000cc`aa65f100 000007fb`01e9478a ntdll!KiUserExceptionDispatch+0x2e
000000cc`aa65f828 000007f7`5be74c8b KERNELBASE!DebugBreak+0x2
000000cc`aa65f830 000007f7`5c059a09 MFCApplication1!CMFCApplication1App::InitInstance+0x2cb
000000cc`aa65f920 000007f7`5c0417fc MFCApplication1!AfxWinMain+0x75
000000cc`aa65f960 000007fb`020d167e MFCApplication1!__tmainCRTStartup+0x148
000000cc`aa65f9a0 000007fb`04ce3501 kernel32!BaseThreadInitThunk+0x1a
000000cc`aa65f9d0 00000000`00000000 ntdll!RtlUserThreadStart+0x1d

Let’s say I want to know what does MFCApplication1!CMFCApplication1App look like at this time. First let us disassemble first few instructions in CMFCApplication1App::InitInstance:

0:000> u MFCApplication1!CMFCApplication1App::InitInstance l10
MFCApplication1!CMFCApplication1App::InitInstance [d:\temp\mfcapplication1\mfcapplication1\mfcapplication1.cpp @ 73]:
000007f7`5be749c0 488bc4          mov     rax,rsp
000007f7`5be749c3 55              push    rbp
000007f7`5be749c4 488d68a1        lea     rbp,[rax-5Fh]
000007f7`5be749c8 4881ece0000000  sub     rsp,0E0h
000007f7`5be749cf 48c745affeffffff mov     qword ptr [rbp-51h],0FFFFFFFFFFFFFFFEh
000007f7`5be749d7 48895810        mov     qword ptr [rax+10h],rbx
000007f7`5be749db 48897018        mov     qword ptr [rax+18h],rsi
000007f7`5be749df 48897820        mov     qword ptr [rax+20h],rdi
000007f7`5be749e3 488b0536f72b00  mov     rax,qword ptr [MFCApplication1!__security_cookie (000007f7`5c134120)]
000007f7`5be749ea 4833c4          xor     rax,rsp
000007f7`5be749ed 48894547        mov     qword ptr [rbp+47h],rax
000007f7`5be749f1 488bf1          mov     rsi,rcx <-- rcx was saved to rsi here
000007f7`5be749f4 c745b708000000  mov     dword ptr [rbp-49h],8
000007f7`5be749fb c745bbff000000  mov     dword ptr [rbp-45h],0FFh
000007f7`5be74a02 488d4db7        lea     rcx,[rbp-49h]
000007f7`5be74a06 ff157c961f00    call    qword ptr [MFCApplication1!_imp_InitCommonControlsEx (000007f7`5c06e088)]

We know that “this” pointer of MFCApplication1!CMFCApplication1App object was passed using rcx, rcx was saved into rsi not far from the prolog, and since the crash was happening inside this function, let’s restore the crash context record and check whether we can get MFCApplication1!CMFCApplication1App pointer from rsi:

0:000> .ecxr
rax=0000000000000001 rbx=0000000000000000 rcx=000007fb04b04aaa
rdx=0000000000000000 rsi=000007f75c1418b0 rdi=000000ccaa860ba0
rip=000007fb01e9478a rsp=000000ccaa65f828 rbp=000000ccaa65f8b9
 r8=000000ccaa65f828  r9=000000ccaa65f8b9 r10=0000000000000000
r11=0000000000000202 r12=000007f75c1418b0 r13=0000000000000000
r14=000007f75c1418b0 r15=00000000ffffffff
iopl=0         nv up ei pl nz na pe nc
cs=0033  ss=002b  ds=002b  es=002b  fs=0053  gs=002b             efl=00000202
KERNELBASE!DebugBreak+0x2:
000007fb`01e9478a cc              int     3

0:000> dt /r MFCApplication1!CMFCApplication1App @rsi
   +0x000 __VFN_table : 0x000007f7`5c0df898 
   =000007f7`5c074d10 classCObject     : CRuntimeClass
      +0x000 m_lpszClassName  : 0x000007f7`5c074d40  "CObject"
      +0x008 m_nObjectSize    : 0n8
      +0x00c m_wSchema        : 0xffff
      +0x010 m_pfnCreateObject : (null) 
      +0x018 m_pBaseClass     : (null) 
      +0x020 m_pNextClass     : (null) 
      +0x028 m_pClassInit     : (null) 
   =000007f7`5c071698 classCCmdTarget  : CRuntimeClass
      +0x000 m_lpszClassName  : 0x000007f7`5c071808  "CCmdTarget"
      +0x008 m_nObjectSize    : 0n64
      +0x00c m_wSchema        : 0xffff
      +0x010 m_pfnCreateObject : (null) 
      +0x018 m_pBaseClass     : 0x000007f7`5c074d10 CRuntimeClass
         +0x000 m_lpszClassName  : 0x000007f7`5c074d40  "CObject"
         +0x008 m_nObjectSize    : 0n8
         +0x00c m_wSchema        : 0xffff
         +0x010 m_pfnCreateObject : (null) 
         +0x018 m_pBaseClass     : (null) 
         +0x020 m_pNextClass     : (null) 
         +0x028 m_pClassInit     : (null) 
      +0x020 m_pNextClass     : (null) 
      +0x028 m_pClassInit     : (null) 
   ...
   +0x180 m_strRegSection  : ATL::CStringT<wchar_t,StrTraitMFC<wchar_t,ATL::ChTraitsCRT<wchar_t> > >
      +0x000 m_pszData        : 0x000000cc`aa851548  "Workspace"
   ...
   +0x1cc m_bHiColorIcons  : 0n1

It looks good. Of course it is completely possible that after the copying “this” pointer from rcx, rsi was changed again before the crash. The only way be sure about it is to disassemble the function from start to the crash point and read the assembly code. I made a bet here that rsi wan’t changed, and I will dump the complete structure to make sure it looks intact.

Another tricky part of 64-bit crash dump analysis is that executable images are as important as symbols. Many times WinDbg needs access to the binaries to be able to fully restore the call stack, there are lots more function information stored in 64-bit binaries than 32-bit binaries. If images are not available, it is still possible to manually construct the stack by walking through output from dps , but it is going to be very painful since rbp is no longer used as frame base pointer.

Use Windows Performance Toolkit to Trackdown Slow Service Start Problem

Terminology

Recommended Readings
Bruce Dawson has an excellent article “Xperf Wait Analysis–Finding Idle Time” which explains how to use Windows Performance Toolkit (WPT) to analyze CPU idle issue. He actually has a series of articles about WPT and I would recommend read all of them if you are new to WPT.

The Issue
Our product has several Windows services, and to reduce boot time impact to the computer, we have an internal guideline that the each service should be started in 0.5 second during boot. One day I received report that a service took more than 1 second to start, they supplied the boot tracing etl file collected by XbootMgr (part of WPT and now replaced by Windows Performance Recorder in Windows Assessment and Deployment Kit). I opened the etl file, loaded symbols, opened services table and filtered to the service in question:

service

  • Container Init Time: this starts when the service process being created by services.exe, ends when the service process calls StartServiceCtrlDispatcher; if there are multiple services sharing same host process, Container Init Time will only be counted for the first service.
  • Service Init Time: this starts when ServiceMain was called, ends when service called SetServiceStatus with status SERVICE_RUNNING.
  • Duration: this is how long service took to start, the value came from “Ended At” minus “Started At”, or “Container Init Time” plus “Service Init Time”.

From the table, we can see that MyService did take 1.079s to start, most of the time was spent in Container Init Time, this means we should make our focus on looking at what the process did at its start up. Fortunately the etl file has ReadyThead (please see this article for details) information and stack-walking was enabled on it, let’s try to find out what is its first thread and what it did.

Zoom Into The Range
Before jumping to ReadyThread, it is recommended to zoom into the time range we are interested in, it not only reduces noises but also improves the data loading and ordering performance. Since I already filtered the data to MyService only, there will be a vertical blue bar in the graph, you can right click on it and select “Zoom” command; alternatively you can right click anywhere on the graph, select “Select Time Range…” and fill in Start and End manually. For example, I set the start and end to same value as MyService start and end time since I will only be interested in data occurred during this time frame:

Select Range

First Thread in MyService.exe
First, I need to find out which is the first thread in MyService.exe. Let’s open ReadThread table, filter selection to MyService.exe, make “SwitchInTime(s)” and “NewThreadId” the second and third column, order data by “SwitchInTime(s)”, and here is what I saw:

First Thread
Again, please see this article for information about the meaning of these columns.

Here not only we know thread 1592 was first thread ran in MyService.exe, but also we noticed something was wrong: thread 1592 was first switched into CPU to be run at 13.207940543s, the next time it was switched in was at 14.080725766s, more than 0.8s after the first switch in. It looks like it was blocked and waiting for some other threads. Let’s re-order the columns to find out which thread awoke (also known as “readied”) thread 1592:

Ready Thread

From the graph above, thread 672 in services.exe is the readying thread of thread 1592. Also from the stack, we can tell that services.exe created thread 1592 (in suspended status) at around 13.207940543s, but didn’t call ResumeThread to let it run until at about 14.080671757s (column ReadyTime(s) in the graph), was it blocked too? Let’s take a look at what’s happening in thread 672 of services.exe.

Thread 672

We saw similar thing was happening in thread 672 as well: after being switched into CPU at 13.208368780s, the next switch in happened at 14.080603959s, again 0.8s after last switch in. Its readying thread was thread 800, in same process. It’s time to take a look at that thread, and it started feeling like falling into a rabbit hole…

Dead End?
I expanded thread 800 in services.exe and following is a snippet of switches from it:

Thread 800

It looks actually pretty normal, the switch in time here was distributed evenly without any noticeable gap from 13.20s all the way down to 14.08s. It looks like a dead end. Or is it?

Switch To CPU Busy Diagnostic
So far we are using ReadyThread to diagnostic CPU idle issue, which are caused by thread was being blocked for too long. If the search reached to a thread without visible delay or blocking, it is time to switch to CPU busy diagnostic model. Most often people are using CPU sampling graph (also known as profiling) for CPU busy issues, here we can also use ReadyThread to investigate which function took most of CPU time, and it is actually more accurate than CPU sampling. Still in ReadyThread table, let’s re-order the columns: remove everything, leave only NewProcess, NewThreadId, NewThreadStack and %CPU Usage, order data by %CPU Usage:

CPU Usage

This is cool. The butterfly stack graph shows that during 13.20s to 14.08s, in thread 800, most of CPU time was spent in function services.exe!RSetServiceObjectSecurity and children functions called by it. Based on stack, we also know that thread 800 was responding to an ALPC message and was modifying a service’s security descriptor. As the result of the modification, it resulted a registry flush which took about 0.8s to finish. This whole process was inside a semaphore object (recall that thread 800 readied thread 672 by calling ntkrnlmp.exe!NtReleaseSemaphore from services.exe!RSetServiceObjectSecurity), which ultimately delayed start of MyService.

Who Did It?
Now we know that the root cause is somebody was modifying service security descriptor during boot, caused registry flush, and delayed other service’s start. The next step is to find out who did it. Based on the knowledge I know how Windows service works, it is sechost.dll that creates worker threads and calls ServiceMain function implemented by the service, and there is a good chance sechost.dll!SetServiceObjectSecurity was called from this thread. Let’s re-order the column in ReadyThread table and see whether we can find somebody called sechost.dll!SetServiceObjectSecurity. It only took a minute for me to find it:

Another Service

AnotherSvc.exe’s thread 1584 made a call to sechost.dll!SetServiceObjectSecurity, it transformed into an ALPC message, handled by thread 800 in services.exe, and once it is done, thread 800 readied thread 1584 as well as another thread 672 in services.exe almost at same time due to the releasing semaphore. Now all the pieces are put together and we got a clear picture of what was going on.

I contacted the team responsible for AnotherSvc.exe and they fixed the issue by moving the call to sechost.dll!SetServiceObjectSecurity at a later stage.

The End
Here we used ReadyThread table (and different columns and orderings) to track down the root cause of the service slow start, across 4 threads in 3 different processes, this is so far the most complicated slow performance issue I have met, and I hope won’t run into another in future.

The strange DHCP “NAK” REPLY

This morning I turned on my notebook and found there was no Internet, output from ipconfig shows the wireless adapter had a self-assigned address instead of one from router. It was working fine yesterday…

I turned on my Nexus 7, it is working, so the problem is not at the wireless access point or the router, the focus should be on the notebook. I launched Network Monitor and captured DHCP packets generated by “ipconfig /renew”, it doesn’t look like the DHCP server was in a good mode:

dhcp

Basically notebook asked for an offer for IP address, DHCP server gave one, notebook requested confirmation, and DHCP server rudely rejected with a NAK reply.

I was suspecting this must has something to do with the static IP address assignment I setup on the router. To be able to use IP address instead of machine name for device-to-device communication, each device will get a fixed address from router based on its mac address. On the notebook, I changed the wireless adapter to use a static IP to be able to login to the router and deleted the static IP rule for the notebook, then changed the wireless adapter back to dynamic IP. After disconnect/re-connect wireless adapter, everything worked fine.

I think I also know what caused this problem. Yesterday I was running a VMware Workstation virtual machine on the notebook, and changed its virtual network adapter from NAT to bridged to do some test. This means it will get IP address from my home-network router, using the notebook’s wireless adapter. Somehow this upsets the DHCP server with static IP rule setup for the notebook. Originally the notebook will get 192.168.1.102 from router by the static IP rule, yesterday the virtual machine was getting 192.168.1.108 after I changed its network adapter to bridged mode. Looks like there is a bug in router firmware, where it gave 192.168.1.108 as offer and then rejected it later as it found out 192.168.1.102 should be given based on notebook’s mac address.

I think I will leave the notebook to get dynamic ip address from DHCP server to avoid this from happening again.