SE Dallas News

12:50 update, Monday, May 22

There remains no established evidence of a data leak.

Municipal Court is undergoing a previously planned upgrade and anticipates reopening Tuesday, May 30.

While Court has been closed since May 3 due to the network outage, City of Dallas residents have been able to mail payments, requests, or documents to court.

All cases will be reset and notices will be mailed. Citation payments or documents due while Municipal Court is closed will be processed after May 30.

_________________________________________________________________________________________________________________

1:20 p.m. update, Friday, May 19

The City of Dallas is aware of a post from what appears to be the Royal ransomware group threatening to release city data. We continue to monitor the situation and maintain there is no evidence or indication that data has been compromised.

Measures to protect data are in place. Information will continue to be provided on DallasCityNews.net as the ongoing investigation of this criminal organization continues.

_________________________________________________________________________________________________________________

2:00 p.m., Thursday May 18 Update

311 functionality is restored including the OurDallas app and website.

_____________________________________________________________________________________________________________________________________________________________________________________

3:00 p.m., Wednesday May 17 Update

Development Services is accepting electronic plan submissions, processing payments, and issuing permits. The next Pop-Up Permit Saturday event is 8 a.m. to 2 p.m. Saturday at the Oak Cliff Municipal Center, 320 East Jefferson Blvd. Dallas, TX 75203.

_______________________________________________________________________________________________________________________________________________________________________________________

3:00 p.m., Tuesday May 16 Update

Due to preparations made in advance and plans followed, restoration continues with a primary focus on public safety followed by resident-facing public services.

Development Services is temporarily unable to process payments; however, plans may still be uploaded online or submitted in-person at the Oak Cliff Municipal Center, 320 E. Jefferson Blvd., 75203, and submitters will be contacted to remit payment when service is restored.

Office of Community Care (OCC) Vital Statistics has limited its ability to issue birth and death records and cannot accept or process online orders. Residents may call OCC at 214-670-3092 or request records from the State’s website: Order Vital Records | DSHS Applications | Texas.gov

___________________________________________________________________________________________________________________________________________________________________

3:00 p.m., Monday, May 15 Update

Due to preparations made in advance and plans followed, restoration continues steadily, with additional services brought back online daily. Given the complexity of checking, cleaning, and restoring interoperability to remaining departmental devices, systems and applications, it will likely take weeks to get back to full functionality.

Public safety remains our highest priority. 911 has continually taken calls to dispatch Dallas Police Department (DPD) and Dallas Fire-Rescue (DFR) to protect and serve Dallas residents.

After public safety, public-facing services are being prioritized to minimize disruption of services to the public. There is still no indication that data from residents, vendors, or employees has been leaked.

1:30 p.m., Friday, May 12 Update

Due to preparations made in advance and plans followed, restoration continues steadily. Public safety remains our highest priority. After public safety, public-facing services are being prioritized to minimize disruption of services to the public. There is still no indication data from residents, vendors or employees has been leaked. Given the complexity of checking, cleaning, and restoring interoperability to remaining departmental devices, systems and applications, it will likely take weeks to get back to full functionality. If there is new restoration of a currently unavailable public-facing service, we will advise immediately. ITS will continue to work through the weekend; however, the next status update will be sent Monday, May 15.

3:30 p.m., Thursday, May 11 Update

Public safety remains our highest priority. 911 has continually taken calls to dispatch Dallas Police Department (DPD) and Dallas Fire-Rescue (DFR) to protect and serve Dallas residents. More info: bit.ly/CODNetworkOutage

There is no indication data from residents, vendors or employees has been leaked.

Development Services is accepting electronic plan submissions, processing payments, and issuing permits. The next Pop-Up Permit Saturday event is 8 a.m. to 2 p.m. Saturday, May 20 at the Oak Cliff Municipal Center, 320 East Jefferson Blvd. Dallas, TX 75203.

Dallas Water Utilities (DWU) payment systems and meter reading software are back online. For questions or assistance please call (214) 651-1441 or visit the City Hall lobby 8 a.m. through 5 p.m. Monday through Friday.

The City Controller can print checks to vendors and payroll is unaffected.

1:30 p.m., Wednesday, May 10 Update

At this time, it does not appear any personal data of employees or residents has been leaked.

911 calls continue to be entered into the Computer Assisted Dispatch (CAD) system.

DPD has been dispatching automatically since Monday and each day gains computer functionality with less reliance on paper backups.

22 DFR stations’ rescues and apparatus [ambulances and trucks] have been cleaned and partial resumption of automatic dispatch at cleaned locations is expected today.

Development Services continues to accept payments and issue permits and ability to receive electronic plan submittals has been restored. The next Pop-Up Permit Saturday event is 8 a.m. to 2 p.m. Saturday, May 20 at the Oak Cliff Municipal Center, 320 East Jefferson Blvd. Dallas, TX 75203.

Dallas Water Utilities (DWU) payment systems are working. Meter reading software is expected to return to service this week. While usage may be estimated this month, subsequent bills will ultimately be updated to reflect actual usage and amount owed (if any) once meter reads are again available. For questions or assistance please call (214) 651-1441 or visit the City Hall lobby 8 a.m. through 5 p.m. Monday through Friday.

3:30 p.m., Tuesday, May 9 Update

The City of Dallas’ preparation for security threats and early response to network outages caused by a cyberattack May 3 have maintained essential City services for residents throughout the week.

911 calls are being entered into the Computer Assisted Dispatch (CAD) system. DPD has been dispatching automatically since Monday and DFR is routing calls for automatic dispatch where available. Progress continues while vehicles aren’t out on calls.

311 encourages residents to call 311 directly while the OurDallas app and website are still limited in functionality.

Dallas Water Utilities (DWU) payment systems are working. Meter reading software is impacted, so usage may be estimated this month, but subsequent bills will ultimately be updated to reflect actual usage and amount owed, if any, once meter reads are again available. For questions or assistance please call (214) 651-1441 or visit the City Hall lobby 8 a.m. through 5 p.m. Monday through Friday.

Municipal Court is open to provide general citation information and accept customer documents in person. However, the Municipal Court cannot accept payments in person, online, or by phone.  There are no court hearings, trials, or jury duty until further notice. All cases scheduled during this outage will be reset. Updated court dates will be mailed to the address the court has on file. Citation payments due while the Municipal Court system is down will be accepted after service is restored. You can also mail in any payments, requests, or documents.

Library – Most public computers are down. Call your location before visiting. WiFi is available for customers bringing their own devices. The catalog is down, but materials can still be checked out manually. Materials cannot be checked in, so customers are asked to hold on to their checked-out items, even if they are due. The library does not charge late fines. Ebooks, e-audiobooks, magazines, movies and music are available through the Overdrive and Hoopla services.

Development Services is currently accepting payments and issuing permits. The next Pop-Up Permit Saturday event is 8 a.m. to 2 p.m. Saturday, May 20 at the Oak Cliff Municipal Center, 320 East Jefferson Blvd. Dallas, TX 75203.

 

4 p.m. update, Monday, May 8

As of Monday, May 8 DallasCityHall.com and DallasPolice.net are back online.

911 and 311 intake and dispatch continue via phone and radio dispatch while Computer Assisted Dispatch (CAD) components including 1900 mobile devices (1600 for DPD and 300 for DFR) and the server routing calls are tested to ensure no reinfection when redeployed. Completion of device cleaning to allow resumption of CAD is anticipated early this week

Please see below for frequently asked questions regarding the incident:

1. Will the City pay the ransom?

• The City is exploring all options to remediate this incident. As this is an ongoing criminal investigation, the City cannot comment on specific details which risk impeding the investigation or exposing vulnerabilities that can be exploited by an attacker.

2. Did the attackers send a ransom note to City printers?

• An image of information allegedly transmitted by an attacker has been published. While some media outlets removed or blurred a URL shown, others did not. Do not attempt to visit this URL as it may pose a threat to the device or network of anyone that does.

3. How did the breach initiate?

• This is an ongoing criminal investigation. The City cannot comment on specific details which risk impeding the investigation or exposing vulnerabilities that can be exploited by an attacker. The most common Ransomware attacks are initiated by exploiting vulnerabilities in software such as weak or default credentials, and social engineering (e.g. phishing) which tricks users into divulging confidential or personal information that may be used for fraudulent purposes.

4. How many devices are affected?

• This is an ongoing criminal investigation. The City cannot comment on specific details which risk impeding the investigation or exposing vulnerabilities that can be exploited by an attacker.

5. What departments are affected and how?

• See https://www.dallascitynews.net/ for detailed updates.

6. What are next steps?

• The City’s IT Department continues to work with the assistance of cybersecurity experts and vendors to review software, servers, and devices to ensure they are uninfected before they are returned to service.

7. Are 9-1-1 dispatchers still having to take down information by hand and share it over a radio to respond?

• As prepared for and practiced in advance, 9-1-1 operators continue to answer and dispatch calls utilizing back up procedures and the City’s public safety radio system. As City staff and contractors review devices, ensure they are secure, and bring them online, Computer Assisted Dispatch (CAD) functionality will increase for DPD, DFR and 311.

8. What do people on deadline to pay citations need to do?

• Citation payments and documents due while the Municipal Court system is down will be accepted after service is restored. Anyone who has a citation to pay or documents due while Municipal Court is closed will be granted an extension to pay or present the documents to the Court without penalty.

9. How long this will take to get resolved?

• The City’s IT Department will continue to work with the assistance of cybersecurity professionals and our system vendors to review software, servers, and devices to ensure they are uninfected and can be returned to service as quickly as possible.

10. Is the prior network outage related to this attack?

• There is no evidence that the network outage occurring on April 19, 2023 is related to this ransomware attack. The network outage of April 19 was caused by a hardware failure. The City and its network services provider AT&T resolved that outage.

11. Has Royal made any demands from the city of Dallas? Has the city learned how Royal was able to gain access to its systems? If so, how did the group compromise those servers? If so, what is the group demanding, and what has been the city’s response?

• This is an ongoing criminal investigation. The City cannot comment on specific details which risk impeding the investigation or exposing vulnerabilities that can be exploited by an attacker.

12. Should residents be concerned that their personal information could be leaked online as a result of this attack? Why or why not?

• At this time the City has no indication that customer information such as billing data or personally identifiable information (PII) has been leaked from City systems or databases. Should this change, the City will notify potentially impacted individuals with information and instructions. Should any individual be contacted by someone claiming to represent the City of Dallas and ask for a payment or personal information, please take note of the number they are calling from and the number they are reaching, then hang up and call the City Department to report.

13. How much is this attack costing the City?

• As this is an ongoing criminal investigation, a determination of financial impact cannot be given at this time.

14. Were social media channels compromised as a result of the outage?

• The City of Dallas social media accounts have not been compromised. Updates will continue to be shared via DallasCityNews.net.

3:10 p.m. update, Sunday, May 7

City staff and vendors continue working throughout this weekend to ensure progress toward service restoration. As services are brought online, updates will be shared via DallasCityNews.net and the City’s broadcast and social media channels accessible from DallasCityHall.com. On Monday, Chief Information Officer Bill Zielinski will brief the Public Safety Committee.

1:30 p.m. update, Saturday, May 6

As of Saturday, May 6, steady, measured progress continues with minimal service disruption. For the public, departmental service updates are unchanged since Friday night.

We are confident we have contained the source of the infection and not seen any new spread. ITS cybersecurity vendor Crowdstrike continues to ensure that any City devices blocked and quarantined to prevent or contain the spread of the virus are clean before they are back in service. Microsoft continues work toward restoration of departmental web pages from backups, but the demand of so much activity on available systems is challenging capacity.

Our priority remains the restoration of public safety functions such as Computer Aided Dispatch (CAD) for 911 and 311, as well as public-facing services including websites and payment and permitting systems. Each device, webpage, and system will be brought back as soon as safely possible to prevent risking any further setback.

On Monday, Chief Information Officer Bill Zielinski will brief the Public Safety Committee.

5 p.m. update, Friday, May 5

The City of Dallas’ preparation for security threats and early response to network outages caused by a cyberattack May 3 have maintained essential City services for residents throughout the week.

Since Wednesday morning, when a group called Royal initiated an illegal attack targeting the City, Dallas’ Information and Technology Services (ITS) department and ITS’ expert cybersecurity vendors have effectively worked nonstop to swiftly isolate a virus and gradually restore service, prioritizing public safety and resident-facing departments. Much progress has been made, but the recovery process is ongoing.

Thanks to heroic teamwork by our first responders, key public safety functions continue as usual despite technical difficulties. 911 and 311 calls are being answered and Dallas Police Department and Dallas Fire-Rescue are being timely dispatched by radio.

For residents with non-emergency needs, 311 is still taking service requests by phone but the OurDallas app and online portal are temporarily unavailable. Another option is service in person at City facilities during regular business hours.

Sanitation collection remains on schedule and disposal sites are operational during regular business hours. If service is missed, please call 311.

Dallas Water Utilities service is unaffected, and disconnections are discontinued until the outage is resolved. Statements may be paid by mail; however, for those who prefer to pay in-person or online, late fees will not be charged for payments that cannot be processed until service is restored. DWU autopay will draft when service is restored. If you have questions or need assistance, please call (214) 651-1441 or walk into the water lobby at City Hall Monday through Friday between 8 a.m. to 5 p.m.

Office of Community Care clinics for Women, Infants and Children (WIC) are open and providing benefits. Vital Statistics is issuing records, but to ensure any records sought are available before you arrive, please call 214-670-3092.

Dallas Public Library branches are open, operational, and can check out media to residents with a library card; however, residents with media due to be returned are asked to hang on to it a little longer. There will be no late fees charged for materials due during the service outage. Digital media is also available via Hoopla and Overdrive. Internet-connected computers may be limited, so users in need of online device access should call ahead to their respective branch.

Dallas Animal Services continues handling adoptions, fosters, rescues, and returns to owners in-person on a case-by-case basis at 1818 N. Westmoreland Road, 75212.

Code Compliance is issuing garage sale permits only in-person at their headquarters at 3112 Canton Street, 75226.

While pages on the City’s website are being restored, Special Events permit requests may be submitted through the following direct links:

• Special Event, Commercial Promoter, and Fair Park Parking License Application
• Street Pole Banner Application
• Commercial Filming
• Neighborhood Market

Development Services can review paper plans for walk-ins at 320 E. Jefferson Blvd., 75203 during regular business hours. However, while Permitting, Public Works, and Zoning application and payment systems are offline, submissions cannot be received or approved. This is a dynamic situation, and patience is appreciated while we focus on expediting full-service restoration.

Municipal Court remains closed Monday, May 8. There will be no court hearings and no trials. Cases scheduled during this outage will be reset, and updates will be mailed. Citation payments and documents due while Municipal Court is closed will be accepted after service is restored.Please note, no one from the City of Dallas will reach out to members of the public to ask for payment in person or by phone. Never give out your password or payment information by phone or through an email link. To protect against cyber threats please install the Dallas Secure app on your iOS or Android device. If you are contacted by someone seeking payment who claims to be from a City of Dallas department, please take note of the number they are calling from and the number they reached you on, then hang up and call the City of Dallas department they claim to be from to report this potential impersonation.

For years the City has monitored to ensure that residents’ and vendors’ information is not leaked. If it were, we would advise the public immediately. At this point, we do not believe that’s the case. Please pardon the inconvenience, thank you again, and stay safe!

4:30 p.m. update, Thursday, May 4

Services to residents continue with some delays and disruption due to the outage. Departmental updates as of 4:30 p.m. are as follows:

ITS isolated the issue and is gradually restoring service prioritizing public safety and resident-facing departments.

DPD and DFR service continues as usual.

311 and 911 calls continue to be received and dispatched.

Dallas Water Utilities is unable to process payments. Disconnections will be discontinued until the outage is resolved.

Vital Statistics is issuing records with limited capacity. Some records may be unavailable, especially from prior to 2005. Residents may contact Vital Records at 214-670-3092.

Women, Infants & Children (WIC) is maintaining normal operations at all clinics and able to issue benefits.

Courts remain closed and cases will be reset; jurors do not need to report for service and notices will be sent by mail.

Code Compliance Services response to service requests may be delayed. Code is currently unable to process Single-Family and Multi-Tenant registrations. Garage sale permits can be issued in-person only at 3112 Canton St.

Dallas Animal Services is responding to injury and emergency requests and non-emergency response is delayed. DAS is handling adoptions, fosters, rescue and return to owners on a case-by-case basis at 1818 N. Westmoreland Rd. during regular business hours.

For Office of Special Events permits please use the following links to login and create application:

• Special Event, Commercial Promoter, and Fair Park Parking License Application
• Street Pole Banner Application
• Commercial Filming
• Neighborhood Market

City Secretary’s Office Open Records Requests will be delayed.

Development Services, Permitting, Public Works, and Zoning applications and payments cannot be received, and permits cannot be issued.

Update as of 10 a.m. May 4.ITS and its vendors continue to work around the clock to contain the outage and restore service, prioritizing public safety and public-facing departments. A group called Royal initiated the attack. Chief Zielinski will brief the Public Safety Committee Monday, May 8.

As of 10 a.m. these are service updates:

• DPD and DFR service to residents is unaffected.
• 911 calls continue to be received and dispatched.
• 311 Calls are being answered but non-emergency service requests may be delayed.
• Courts are closed and LiveChat is inaccessible. All cases will be reset; jurors do not need to report for service and notices will be sent by mail.
• CSO – Saturday’s election is unaffected; Dallas County will share official information including results. Meeting notices are being posted and meetings may be viewed at dallascityhall.webex.com, dallascitynews.net/watch-live, and Spectrum channels 16 & 95 and AT&T U-verse at 99. Contracts may be delayed.
• Dallas Public Library All branches are open and in-person checkouts continue; online materials are currently unavailable.
• DWU Billing is unaffected; meter reading will be delayed. Only IVR can take credit card payments. Disconnections will be discontinued until the outage is resolved.

City Manager T.C. Broadnax has released the following statement:

“Since City of Dallas’ Information and Technology Services detected a cyber threat Wednesday morning, employees have been hard at work to contain the issue and ensure continued service to our residents. While the source of the outage is still under investigation, I am optimistic that the risk is contained. For those departments affected, emergency plans prepared and practiced in advance are paying off. We apologize for any inconvenience and thank residents for their understanding as we continue to work around the clock until this issue is addressed. For updates, please keep an eye on dallascitynews.net.”

Update as of 5 p.m. May 3.

ITS will continue working around the clock to identify the source of the outage and isolate impacted devices. Departmental service to residents continues with minimal disruption as follows.

• DPD & DFR: Service to residents is unaffected although the home page is offline.
• 911: Calls continue to be received and dispatched
• 311: Phone IVR is fully functional. 311 agents are accepting phone calls and dispatching Service Requests.
• Courts: Court was closed Wednesday and will also be closed on Thursday. All cases will be reset and notices will be sent by mail.
• DWU: Payment may be processed via IVR. Online payment processing may be delayed but no disconnections will be processed until the outage is resolved.

Currently less than 200 of the City’s thousands of devices are impacted, but if any City device is at risk, it will be quarantined and blocked by ITS. For compromised machines, restoration will prioritize public safety, anything public-facing, then all other departments.

DallasCityNews.net will be updated at least once a day until the incident is resolved.

Wednesday morning, the City’s security monitoring tools notified our Security Operations Center (SOC) that a likely ransomware attack had been launched within our environment.

Subsequently, the City has confirmed that a number of servers have been compromised with ransomware, impacting several functional areas, including the Dallas Police Department Website. The City team, along with its vendors, are actively working to isolate the ransomware to prevent its spread, to remove the ransomware from infected servers, and to restore any services currently impacted. The Mayor and City Council was notified of the incident pursuant to the City’s Incident Response Plan (IRP).

The City is currently working to assess the complete impact, but at this time, the impact on the delivery of City services to its residents is limited. Should a resident experience a problem with a particular City service, they should contact 311. For emergencies, they should contact 911.

Original source can be found here.