Modern software applications are complex, highly integrated collections of components, authored by dozens or even hundreds of individuals, and the rise of open source has taken this complexity to the next level. As an end-user, how well do you understand what a piece of software is *actually* doing, under the hood? Is your favorite string padding library making outbound network calls? Is your media player creating local user accounts during installation? In this session we’ll explore how we can use both static and runtime analysis techniques to identify security characteristics of software applications. We’ll share our experiences building two open source tools, Attack Surface Analyzer and AppInspector, and describe their value to and place in a modern DevOps pipeline.