City computer hacking was just a test

On the morning of Aug. 13, armed men disguised as janitors strolled into Las Vegas City Hall, made their way to the city's data center and started hacking into computer servers.

The hackers used visitor ID passes to gain access to the digital nerve center and keyloggers — sort of like wiretaps that record computer keyboard strokes — to take over a city email address.

It didn't take information technology staffers long to catch on. Some frantically swept the building for hacking bugs and stolen iPads or laptops, while others scanned servers for potential damage to city email passwords.

Within six hours, they had identified the internal security breach and shut down access to the city's email servers.

A single phishing email sent from a compromised city inbox reached 12 employees before the city managed to root out the intruders.

It was only later revealed that the whole thing was a test — a practice hack conducted by contracted private security consultants and paid for by the city.

But longtime Chief Information Officer and Information Technologies Director Joseph Marcella didn't know that. Neither did his boss — Chief Financial Officer Mark Vincent — nor any City Council members.

So IT staffers continued their investigation into the incident, calling the city's department of Detention and Enforcement, which handles security at city facilities, and the FBI, following protocol.

The feds hadn't responded to the hack by the time Marcella said City Manager Betsy Fretwell — or someone else privy to the practice hack — ordered his investigative team to "stand down."

He had been asking around about the identity of the hackers, what vulnerabilities they had encountered and which officials were in on the exercise.

Four days later, Marcella found himself out of a job — the lone casualty in a botched secret hacking operation that city leaders, staffers and security experts say should have been handled much differently.

Less than a week after his exit, Fretwell went on a city-owned TV program to congratulate Marcella on his retirement and applaud the IT department's "astounding" progress under his tutelage.

Departure makes waves

Marcella, the 67-year-old former director who started his tenure with the city in 1997, hadn't planned to retire from his post for several more months, during which time he hoped to oversee the arrival of the city's next technology chief.

But that was before his reaction to the hack "really ticked off" Fretwell, he said.

Marcella said private citizens' data was not breached in the course of the secret security test. But that doesn't mean it wasn't at risk, one of the reasons Marcella said he was "livid" when his team was told to stop its investigation.

"I don't want to cast aspersions on the city manager," Marcella said. "What she did was probably an emotional thing.

"Maybe she thought I was undermining her. Maybe she thought if I had carried the cyberanalysis further it would have become public, and that would have been an embarrassment. A dozen different things could have been on her mind."

Fretwell said Marcella "separated" from the city via retirement, but declined to comment on the circumstances surrounding his exit. She said he received the standard separation severance pay awarded to all outgoing executive administrators.

She cited security concerns in refusing to confirm or deny whether a hack had ever taken place at City Hall.

CFO Vincent, Marcella's former boss, confirmed the hack had taken place, though he said there was "nothing unusual" about Marcella's departure from the city.

Another account of his exit — relayed by a city staffer who declined to be identified for fear of retribution — hewed closer to Marcella's version.

"I don't think Betsy's response was proportionate," the staffer said. "Everybody was shocked at the way this was handled, just as shocked as Joe was.

"Many people were affronted, not just here but in Henderson and elsewhere, because of Joe's reputation and how well he's respected."

Marcella said he's still not sure why Fretwell didn't tell someone on his team about the hacking test, a precaution experts say is observed throughout the cybersecurity world as a means of ensuring an effective, proportionate response to tech security threats.

Experts identify irregularities

Todd Sander, a former municipal CIO and executive director of the Center for Digital Government, a California-based research and advisory institute, said it was "rather strange" city officials didn't let a single IT staffer in on the ruse.

Perhaps no less strange: the decision to have contractors conduct a "physical" examination of the city's tech infrastructure — by allowing armed contractors to test City Hall's key card readers, video cameras and other security equipment — at the same time as they carried out a software security check.

"In my experience, they're usually done separately," Sander said. "That's the logical thing to do."

The test was performed less than two months after a hacker gained undetected access to the city of Henderson's servers for nine days.

Marcella suspects someone privy to the attack, perhaps wary of Henderson's example, acted on a security contractor's advice to keep the test as "pure," or secretive, as possible.

He said Universal Security Specialists Inc., a security consulting firm run by former Nevada Attorney General Catherine Cortez Masto's husband, Paul Masto, were the ones hired to do the job.

That company did not return requests for comment.

The city cited security concerns in refusing to hand over requested copies of two contracts, totaling $30,000, detailing the scope of the firm's work for the city.

The Review-Journal is still awaiting thousands of requested emails sent and received by Marcella and Fretwell in the run-up to the former IT director's sudden departure.

Experts said pressuring officials to keep the hack a secret might have been a good way for vendors to rack up additional consulting fees, but it's not the best way to identify and patch security glitches.

RJ Robinson, a Loma Linda, Calif.-based cybersecurity expert, said contractors shouldn't have allowed IT staffers to carry on as if faced with an ongoing clandestine attack.

In his view, if anyone should have been let go, it was the firm hired to do the work.

"The security contractor overstepped," he said. "They didn't do a very good job. These IT guys are doing their job. They're reacting to what they think is a real-world situation."

Exit questioned

The haze still hasn't cleared around the question of what city officials knew about the hack and when they knew it.

Vincent said Detention and Enforcement — the department Marcella said relayed the message to shut down the hacking investigation — was in on the cybersecurity test, along with the city manager's office.

Department chief Michele Freeman did not return requests for comment.

Freeman's boss, Deputy City Manager Orlando Sanchez, also failed to respond to requests for comment, as did City Auditor Radford Snelding.

Las Vegas Mayor Carolyn Goodman and all six City Council members said they had no prior knowledge of the move.

More than one said they plan to find out more about it, including three-term City Councilwoman Lois Tarkanian.

"This bothers me," Tarkanian said of Marcella's departure. "It's a hard way to lose a job, especially when you have someone that's outstanding at their work.

"It could have been handled differently. ... Council people were among the last to know, is my understanding."

Marcella has managed to keep an even keel about the incident.

He isn't disgruntled. He doesn't plan to sue. He's working with a consulting firm and weighing additional job offers.

The only thing that still nags at him is what he said Fretwell told him on his way out the door:

"All she kept saying is 'I'm very disappointed in you,' " Marcella recalled. "How could she be disappointed after 18 years, doing as well as we've done?

"I've had a stellar career, never been canned in my life. ... I didn't do anything wrong, I don't think."

Contact James DeHaven at jdehaven@reviewjournal.com or 702-477-3839. Find him on Twitter: @JamesDeHaven