Last week, one of our data sources stopped functioning for unknown reasons. We opened a trouble ticket, and were immediately told that the issue was with a server migration that was in process, and it would stabilize "in a few weeks".

Well, this wasn't a good outcome considering that the data was "mission critical" to one of the customers, so we experimented and devised a way to get the data, albeit by requesting it multiple times until a working server was found.

Less than 12 hours after we put the "fix" in place, the last of the working servers was taken off-line.

After a marathon programming session, we discovered the real cause - the data was being sent, but it was in invalid XML documents. Because the XML failed validation, the transfer program had no source of data. The XML contained non-UTF characters embedded in an allegedly UTF-8 XML document.

We added code to our application to strip out the non-UTF data and make the XML valid again, and canceled the trouble ticket with a message saying, "We found what you broke. We've worked around it."

"Oh, did you find that non-UTF characters were being inserted? We have programmers working on fixing that."

"And you couldn't tell us this 5 days ago, when you sent us off down a rabbit hole to a fix that almost worked, until you broke something else?"

Still no answer to that question.

Root cause is they switched from a database that stripped all non-ASCII character from the data to one that had lots of marketing-added symbols that were non-ASCII and non-UTF (actually using a Windows codeset), and didn't realize that the data was different. Then they doubled-down by not stopping the migration until they fixed the problem, AND "neglected" to inform anyone of the problem...

And they chastise us for not doing things in the proper "Enterprise" model, like they do...