In January 2010, POST staff reported the Testing Management and Assessment System’s (TMAS) inability to adequately secure and protect tests to the State Financial Integrity and State Manager’s Accountability (FISMA) audit. The audit subsequently ranked this issue as the second highest risk to POST. After analysis, POST staff concluded that the TMAS system is technically and functionally obsolete, and cannot be corrected through product updates or modifications.

In June 2010, the Commission directed staff to contract for the completion of an Information Technology Feasibility Study Report (IT FSR) and Information Technology Procurement Plan (ITPP) to replace TMAS with a modern, enhanced testing system. The proposed system is to be capable of securing POST tests, as well as supporting POST’s current testing and audit requirements.

The Commission authorized the Executive Director to contract for FSR and ITPP authoring assistance because the required effort exceeded POST staff’s resources. In January 2011, following a competitive bid process, staff contracted with Hubbert Systems Consulting to assist in preparing these analyses. The contract procurement and deliverables were performed in compliance with Sections 4920-4942 of the State Administrative Manual. 

Issues with the current TMAS testing system may broadly be split into two areas: security and functionality:

Security:

The TMAS system was developed eight years ago by a vendor following a competitive bid process. The system uses Internet-based technology to deliver POST tests to proctored testing locations throughout California. The TMAS system has been updated occasionally to address some functional and usability issues. However, the system’s security model has not changed since implementation. 

As Internet technology has matured over the past decade, security measures have also improved: PC lockdown procedures, activity logging, intrusion detection, and encryption are much more robust than they were when TMAS was designed. During the same period, the tools and knowledge needed to compromise secure systems have also become more widely available. The TMAS system was not designed to address these newer and more common technical threats, or to accommodate new security technologies as they are developed.

POST staff have performed significant market research on modern configurable off-the-shelf (COTS) testing software, and concluded that modern products offer much-improved security features and ease of set-up and use. If this item is approved, a competitive bidding process based on comprehensive POST requirements will be performed to select the product to best fit POST’s needs.

Functionality:

The TMAS system’s sole focus is on the delivery of academy written tests. Due to its monolithic, non-modular design, TMAS is not capable of accommodating other POST tests, such as the entry-level law enforcement test, public safety dispatcher test, and academy skills and scenario tests. 

TMAS is also not capable of using modern data-exchange techniques to share data with reporting and statistical analysis packages. For example, exporting several years of a specific test’s results from a selected group of academies, for trend analysis and test development purposes, is not possible. Similarly, POST cannot currently export test data into graphing or pattern-recognizing analysis tools to detect and prevent cheating. These shortcomings hinder POST staff’s ability to develop new learning techniques, as well as to secure test assets.

Market research indicates that modern testing software is flexible and highly modular, allowing many test types to be integrated into the system over time. Additionally, standards for data interchange, such as “Sharable Content Object Reference Model” (SCORM) and configurable Web services, allow modern systems to exchange data relatively easily with other systems, reporting tools, and statistical analysis packages.

Staff recommends that the Commission authorize the Executive Director to contract for products and services necessary for the acquisition, configuration, implementation, and verification of software to replace POST’s current automated testing system in an amount not to exceed $2,739,560.