An Isolation Approach for Secure and Flexible Binary Authentication

Abstract

In recent years, many computer security issues have come to be regarded as significant social problems globally. Malicious programs such as viruses, Worms and trojans, also called malware have especially serious effects on the IT industry. Malware can be installed on a system via different sources, and executed to perform malicious acts for nefarious purposes. To prevent execution of these undesirable programs, many binary authentication schemes have been proposed. However, they have so far proved insufficient as they have security deficiencies and are not flexible and pragmatic enough.In this thesis, we present a run-time flexible binary authentication system for a virtual machine that can successfully authenticate running binaries. Moreover, this system can analyze binaries at runtime in cases when the signatures for the binaries are absent. The proposed system is implemented on a hypervisor, thus attaining an isolated security model. Running binaries on a guest virtual machine (VM) are authenticated using a cryptography hash and analyzed by security tools within a privileged domain called dom0. Our system guarantees that only authorized binaries can be run, and it identifies unknown binaries, dynamically generated codes, and self-modifying codes.