After the New York City Council’s 10-hour hearing on artificial intelligence, I’m not sure we’re any closer to answering exactly how dangerous these models are or what companies are doing to protect us.
I attended because I was curious how these companies would approach public accountability in, what felt like, a high-stakes situation. But I walked away with the sense that, while each company said they are doing as much as they can to ensure their models are developed safely, we are barreling toward a future that most of us had little say in determining. And if Jacob Coxon is right, that future might spell the end of humanity.
Let’s dive in.
Jared Giles
Founder, SquareOne
[email protected]
Individual Gain vs. Global Risk
On Monday, the NYC Council heard testimony on the risks and benefits of artificial intelligence, focusing on strategies to establish regulatory guardrails for American tech companies. AI whistleblowers along with representatives from Anthropic, OpenAI, Google, and Meta testified under oath, and additional statements were heard from city officials, think tanks, and public advocates. It was a chance for the public to finally hear each company be held accountable for what they’re building, and be challenged to answer how they plan to keep us safe. But once I saw that the AI company representatives all chose to attend virtually, I grew skeptical that their testimony would lead to any new or worthwhile information.
Throughout the hearing, there was a lot of back and forth about safety and transparency. If AI might kill us, what do we need to do now to prevent that outcome? The conversation was often prefaced with the monumental gain that artificial intelligence can have on our world, while stressing that companies should build their own safety protocols, which they all say they are doing, alongside local, state, and federal regulation to address the gamut of risks ranging from data center land use to job displacement to AI literacy. It is paramount that we address each of those issues in service of that brighter future, but I have yet to hear a clear articulation of exactly what that future should look like.
When experts talk about the future of AI and its benefits, the discussion usually involves some smattering of the economic, social, and civic benefits that may be derived from the technology, including potential medical advancements and productivity gains. But we have yet to see those universal gains come to fruition. Right now AI's benefits are mostly located at the individual level. For example, during the hearing the representative from Meta hailed Muse, Meta’s new personal AI assistant, as a boon for small businesses. Others have discussed the possibilities AI has opened up for entrepreneurship, tutoring, translation, and many other tasks that a person may have trouble doing on their own.
And while it is true that these are all positive outcomes, the benefits are only experienced by a few. That is to say, as it stands, artificial intelligence is being marketed as what it can do for you, and how you can use it to improve your life or your business, without regard for how it will impact our society and world writ large.
During the hearing, the whistleblowers acknowledged that there is potential for a bright future, but that the direction we're heading in is more likely to go the other way.
OpenAI CEO Sam Altman said recently that “we believe that the world should accept some bad things happening for the benefits of this technology and people having the agency [to use AI widely].” Aside from the incredibly selfish undertone of his words, Sam is highlighting the problem these AI conversations have yet to solve: who decides what our future should look like? Who determines what “bad things” are acceptable for some unknowable benefit? Does that benefit apply to those who are at the forefront of artificial intelligence, or should it be ubiquitous?
We, particularly in the United States, are often taught to think only of ourselves. How can I use this technology to advance my business, to help my family, or to accumulate more power and wealth for myself? We understand there is a global risk but we prioritize and elevate individual gain. So what can be done to curtail these risks for the greater good of everyone?
We need to articulate what that benevolent future should look like. And because the answer is not coming from the federal government or any one corporation, it falls on states and cities to ensure their citizens are taken care of as technology advances beyond our understanding.
AI can be a collective good, and I would like to believe Anthropic CEO Dario Amodei when he said that “most people are underestimating how radical the upside of AI could be.” But if we fail to articulate exactly what that upside should be, and the CEOs of these companies are left to decide for us, I think we are more likely to see a future in which the AI race ends with one of them, a tyrant or oligarch, ruling over a fiefdom that has unencumbered access to all of our lives.
National Spotlight: New York and California Lead with Regulation
With no movement on federal AI legislation, New York and California are stepping in with first-in-the-nation statutory guardrails targeting AI safety, workplace automation, and transparency.
The Problem: The absence of a unified federal framework leaves frontier AI developers and tech platforms operating under a patchwork of voluntary safety commitments—leaving public institutions, workers, and consumers exposed to unvetted models, automated decision systems, and systemic risk.
The Intervention: New York enacted the Responsible AI Safety and Education (RAISE) Act, requiring large AI developers to publish detailed risk mitigation frameworks, submit to third-party safety audits, and report critical safety incidents to state regulators within 72 hours. Simultaneously, California rolled out a legislative package, including SB 1000 (expanding provenance and disclosure rules for generative AI systems), SB 947 (prohibiting employers from relying solely on automated decision systems for discipline or termination), and AB 1405 / SB 813 (establishing state registration and standards for independent AI auditors).
The Takeaway: State legislation is shifting AI governance from self-policing to enforceable statutory compliance. By codifying required risk disclosures, workplace protections, and third-party verification systems, New York and California are building the blueprint for national AI regulation.
ICYMI
[WATCH] NYC Council: Full recording of the hearing on AI safeguards.
[READ] NYC Council: Details on the 10 bills proposed by NYC Council Members to build guardrails around AI, including protecting city employees, requiring third-party validation for the sale or deployment of new models, and removing AI deep fakes of elected officials.
[WATCH] The Diary of a CEO: Steven interviews Daniel Kokotajlo (ex-OpenAI who testified to NYC Council) about AI model development and what’s really happening inside of AI companies.
Stay connected and explore our latest work at SquareOne.