In the context of the software supply-chain, specifically the open source ecosystem that provides raw resources, to many, if not all enterprises, the notion of assurance is applicable for those responsible for the acquisition, receipt and integration of the components/material. This assurance includes notions of trust and trustworthiness, as well as predictability (i.e. it does what it says it is going to do – and nothing else). According to some organizations assurance includes conformance (DHS). Ion Channel would like to include compliance as another factor of assurance that enables risk management and start the dialog.
Conformance can play a part in the risk assessment associated with a resource for sure. If we accept an understanding that conformance is the gauge associated with how well a piece of software meets a specification – we then also have to include that standard/specification into the risk analysis. Questions arise such as: “who manages the specification?” or “is the specification well accepted by a greater community?” or “does the specification meet my requirements?”. If we qualify the specification, conformance becomes a valid risk management attribute. The value may be that alternatives (other software that also implement against the specifications) are available, or that it simply provides a test of the implementation against a known validation source (test kits). Or it might just be the case that a developer has taken the time to provide a solution that meets a specification – and the perceived value is that creates a higher level of quality – and potentially less risk.
Compliance on the other hand is a little more objective, at least after the specification. Compliance is the process(es) of testing a subject against a well-defined policy, or set of policies – the result being pass or fail. Risk management for non-compliance is subjective – a result of the policy writers’ determination of compliance level is an acceptable risk. The completeness and robustness of the policies define what is an acceptable tolerance. For example, various static analysis criteria may be less valuable than determining known vulnerabilities – so policy and compliance may dictate ‘no known vulnerabilities of high or critical status’. While static analysis may provide insight into quality, and thus inherent security/tolerance for change, it might not be a critical path by itself. It makes more sense to treat subjective attributes as aggregates in a compliance framework, or even use them in terms of trends (e.g. ratio of transitive to declared dependencies or identifying a decrease in code coverage over time.) A software project that has a decreasing amount of test coverage, might be non-compliant – rather than attempting to specify a value to associate with a compliance check/test.
Conformance and compliance can be used within the boundaries of risk definition, monitoring, and mitigation/remediation. Both can be as data points to actively drive and support policies, enabling better decisions about software sources, projects and resources (initially, and over time). In addition conformance and compliance can regulate ingest/usage in enterprise continuous integration and delivery practices. Having a framework in place which provides processes to evaluate assurance is just as important as the policies. The framework provides the possibility of continuously reviewing conformance and compliance – promoting constant situational awareness and opportunities for active risk assessment and management.