Existing law requires, among other things related to ensuring the safety of companion chatbots, an operator to prevent a companion chatbot on its companion chatbot platform from engaging with users unless the operator maintains a protocol for preventing the production of suicidal ideation, suicide, or self-harm content to the user, as specified. This bill, the Preventing AI User Self Endangerment (PAUSE) Act, would require an operator to adopt and make publicly available a policy governing its protocol for identifying and responding to credible crisis expressions and, for each companion chatbot an operator makes available to users in this state, implement a system for monitoring and detecting credible crisis expressions in user conversations with companion chatbots. The bill would require, if the monitoring system detects a credible crisis expression, the operator to take certain actions, including commence a crisis interruption pause, as specified. The bill would define "credible crisis expression" to mean a statement by a user of a companion chatbot that reasonably indicates, as determined through contextual analysis rather than keyword detection alone, intent to harm the user or others. This bill would require an operator of a companion chatbot to document certain information related to credible crisis expressions and crisis interruption pauses and, beginning January 1, 2028, annually report that information to the Office of Suicide Prevention. The bill would provide for its enforcement, as specified.
Existing law establishes the Government Operations Agency, which consists of several state entities, including, among others, the State Personnel Board, the Department of General Services, and the Office of Administrative Law. Under existing law, the Government Operations Agency is under the direction of an executive officer known as the Secretary of Government Operations, who is appointed by, and holds office at the pleasure of, the Governor, subject to confirmation by the Senate. This bill would require the secretary, by July 1, 2026, to appoint a mental health and artificial intelligence working group, as specified, that would evaluate certain issues to determine the role of artificial intelligence in mental health settings. The bill would require the working group to take input from various stakeholder groups, including health organizations and academic institutions, and conduct at least 3 public meetings. The bill would require the working group to produce a report of its findings to the Legislature by July 1, 2028, and issue a followup report by January 1, 2030, as specified. The bill would repeal its provisions on July 1, 2031.
Existing law establishes various healing arts boards within the Department of Consumer Affairs that license and regulate various healing arts licensees. Existing laws, including, among others, the Medical Practice Act and the Dental Practice Act, make it a crime for a person who is not licensed as a specified health care professional to use certain words, letters, and phrases or any other terms that imply that they are authorized to practice that profession. Existing law requires, with certain exemptions, a health facility, clinic, physician's office, or office of a group practice that uses generative artificial intelligence, as defined, to generate written or verbal patient communications pertaining to patient clinical information, as defined, to ensure that those communications include both (1) a disclaimer that indicates to the patient that a communication was generated by generative artificial intelligence, as specified, and (2) clear instructions describing how a patient may contact a human health care provider, employee, or other appropriate person. Existing law provides that a violation of these provisions by a physician shall be subject to the jurisdiction of the Medical Board of California or the Osteopathic Medical Board of California, as appropriate. This bill would make provisions of law that prohibit the use of specified terms, letters, or phrases to falsely indicate or imply possession of a license or certificate to practice a health care profession, as defined, enforceable against an entity who develops or deploys artificial intelligence (AI) or generative artificial intelligence (GenAI) technology that uses one or more of those terms, letters, or phrases in its advertising or functionality. The bill would prohibit the use by AI or GenAI technology of certain terms, letters, or phrases that indicate or imply that the advice, care, reports, or assessments being provided through AI or GenAI is being provided by a natural person with the appropriated health care license or certificate. This bill would make a violation of these provisions subject to the jurisdiction of the appropriate health care profession board, and would make each use of a prohibited term, letter, or phrase punishable as a separate violation. By expanding the scope of existing crimes, this bill would impose a state-mandated local program. The California Constitution requires the state to reimburse local agencies and school districts for certain costs mandated by the state. Statutory provisions establish procedures for making that reimbursement. This bill would provide that no reimbursement is required by this act for a specified reason.
