For the last several months, a city at the heart of Silicon Valley has been training artificial intelligence to recognize tents and cars with people living inside in what experts believe is the first experiment of its kind in the United States.
Last July, San Jose issued an open invitation to technology companies to mount cameras on a municipal vehicle that began periodically driving through the city’s district 10 in December, collecting footage of the streets and public spaces. The images are fed into computer vision software and used to train the companies’ algorithms to detect the unwanted objects, according to interviews and documents the Guardian obtained through public records requests.
Some of the capabilities the pilot project is pursuing – such as identifying potholes and cars parked in bus lanes – are already in place in other cities. But San Jose’s foray into automated surveillance of homelessness is the first of its kind in the country, according to city officials and national housing advocates. Local outreach workers, who were previously not aware of the experiment, worry the technology will be used to punish and push out San Jose’s unhoused residents.
City employees are driving a single camera-equipped vehicle through sections of district 10 “every couple weeks”, said Khaled Tawfik, director of the San Jose information technology department. The city sends the training footage to participating companies, which include Ash Sensors, Sensen.AI, Xloop Digital, Blue Dome Technologies and CityRover.
Some of the areas in district 10 targeted by the pilot, such as Santa Teresa Boulevard, are places where unhoused people congregate, sometimes with the city’s encouragement. The light rail station on Santa Teresa Boulevard, for example, is home to the city’s only designated safe parking location for RVs, often used as homes.
There’s no set end date for the pilot phase of the project, Tawfik said in an interview, and as the models improve he believes the target objects could expand to include lost cats and dogs, parking violations and overgrown trees.
“If the City were to productionize this technology, we envision the cameras to be on our fleet motor pool vehicles that regularly drive throughout city limits,” a city employee wrote in a 22 January email.
Ken Salsman, chief technology officer for Ash Sensors, said his company, which specializes in sensors that monitor the structural health of buildings, had not explored homelessness detection before learning of San Jose’s pilot. The experiment provided an opportunity to create potentially marketable technologies by solving challenging computer vision problems, such as distinguishing an empty RV parked outside a home from an RV that is a home. He said the company was training its algorithms to detect proxy signs of habitation.
“Are the windows covered inside the vehicle? Are there towels to provide privacy? Is there trash outside the vehicle, suggesting they’re using food and having trouble getting rid of the waste?” Salsman said. He added that successful detection of lived-in vehicles would probably require frequent scanning of city streets in order to establish whether the vehicles have moved.
A report from the company Sensen.AI shows that its system detected 10 lived-in vehicles in footage collected from two streets on 8 February. Several of the vehicles pictured in the report have tarps spread across windows or rolled up and tied to them. Another has traffic cones next to it. Sensen.AI did not respond to a request for comment.
Tawfik said the goal of the pilot was to encourage companies to build algorithmic models that could detect a variety of different objects from car-mounted cameras with at least 70% accuracy. The participating companies are currently detecting lived-in RVs with between 70 and 75% accuracy, he said, but the accuracy for lived-in cars is still far lower: between 10 and 15%. City staff are following the route of the camera-equipped car and confirming that the vehicles are occupied.
‘We’re not detecting folks. We’re detecting encampments’
City documents state that, in addition to accuracy, one of the main metrics the AI systems will be assessed on is their ability to preserve the privacy of people captured on camera – for example, by blurring faces and license plates. Tawfik said that the city did not “capture or retain images of individuals” through the pilot and that “the data is intended for [the city’s housing and parks departments] to provide services”.
The data use policy for the pilot states that the footage cannot be actively monitored for law enforcement purposes, but that police may request access to previously stored footage.
“We’re not detecting folks,” Tawfik said. “We’re detecting encampments. So the interest is not identifying people because that will be a violation of privacy.” However, in its report identifying lived-in vehicles, Sensen.AI wrote that its system included optical character recognition of the vehicles’ license plate numbers.
San Jose sits at the heart of Silicon Valley. Photograph: StellaMc/Getty Images/iStockphoto
Tawfik said San Jose had delayed its release of a citywide AI policy in part to allow the department to examine its proposed guardrails through the lens of the object detection pilot.
Residents have complained to the city’s 311 phone line about homeless encampments 914 times so far in 2024. They reported illegal dumping 6,247 times, graffiti 5,666 times, and potholes 769 times last year. The goal of the surveillance pilot is to address these complaints more efficiently, according to Tawfik.
According to Tawfik, the city’s response might include sending outreach workers to visit a single tent before it can grow into an encampment, he said. The San Jose housing department and non-profits providing aid to unhoused people said they had not been involved in the pilot.
“Our ability to help the individuals directly is not really part of the pilot,” Tawfik said. “We’re still learning what can be done. And then once the program is mature, then we can look at the data and see what makes sense.”
That approach worries people like Thomas Knight, who was formerly unhoused and now serves as executive member of the Lived Experience Advisory Board of Silicon Valley. The group, made up of dozens of current and formerly unhoused people, has recently been fighting a policy proposed last August by the San Jose mayor, Matt Mahan, that would allow police to tow and impound lived-in vehicles near schools.
“If their whole purpose is to better provide responses to calls to 311, then that means that this computer system is going to identify tents and lived-in vehicles that are in places that the city has deemed they shouldn’t be,” Knight said. “The truth is, the only people you’re going to be able to give [that data] to to fix the issue is the police department.”
San Jose is one of the least affordable housing markets in the country. In order to afford the average effective monthly rent for a one-bedroom apartment in the city, a renter would have to earn $96,000 a year, according to the latest available data. The city’s unhoused population has grown from approximately 4,200 people in 2009 to more than 6,200 in 2023. More than two-thirds of those people are living outdoors and in vehicles rather than the city’s overwhelmed shelter system.
Amid a lack of temporary shelter beds and permanent affordable housing, San Jose officials have cracked down on tent encampments and people living in cars and RVs. Housing advocates fear that identification of encampments by roving AI would add to those efforts.
“The approach to homelessness is to treat unhoused people as blight consistent with trash or graffiti,” said Tristia Bauman, directing attorney for housing at the non-profit Law Foundation of Silicon Valley.
Last fall, the city cleared dozens of people out of tents and vehicles along a half-mile stretch of the downtown Guadalupe River trail and then announced plans for a “no return zone”. This year, police distributed 72-hour notices ordering people to leave a nearby encampment in Columbus Park in order to clear space for the opening of a five-acre dog park.
San Jose: a technological bellwether
In addition to providing a training ground for new algorithms, San Jose’s position as a national leader on government procurement of technology means that its experiment with surveilling encampments could influence whether and how other cities adopt similar detection systems. The city’s IT department is leading a national coalition of more than 150 municipal agencies working to develop policies for “responsible and purposeful” deployment of AI technologies in the public sector.
Tawfik said his staff had discussed the object detection pilot with coalition members and hoped that other agencies would participate in the review process. Companies participating in the pilot have also expressed interest in the “scalability” potential the coalition represents, according to emails they sent to IT department staff.
“As we see more interest from other cities to participate, we’re sharing notes and hopefully that advances the program faster,” Tawfik said.
Knight, from the Advisory Board, said the city’s focus on perfecting a technological solution ignored the root cause of the housing crisis in San Jose.
“If you have no place to put people, it’s pretty much useless,” he said.