We have a configuration that is (supposed to) ensure that mAirList continues to function in live-present mode if the studio drops off the network or the database server is bounced for maintenance. As we approach cutting over to mAirList in 3 of our studios I’ve been testing the robustness of our configuration.
All resources we need for live presentation–playlists and audio files–are hosted on the same PC as mAirList or on an in-studio file server connected to the same in-studio switch as mAirList. All items that are known to the database have in-studio cache copies and I have independently tested that the cached file is used if the file referred to by the database (or its storage) becomes unavailable.
If I disrupt the connection between the studio switch and the core switch, most-but-not-all of the time everything keeps working as desired. Occasionally however the UI freezes (animations stop) and audio playback stops.
I’ve noticed that the system log accumulates messages like this every 17 seconds for as long as the database is inaccessible:
2019-05-24 18:28:13 Error Error connecting to Database: [FireDAC][Phys][PG][libpq] could not translate host name "mairlist-db.rrr.internal" to address: Unknown host
Every 5 minutes a burst of 25 messages like this are generated in the space of 1 second. I’m not sure what precipitates them. Their appearance doesn’t seem implicated in the system lockup.
2019-05-24 18:27:56 Error Error creating playlist cache file C:\mAirList\PlaylistCache\{C4A9766D-A2E2-46E2-9AB4-7D382DAEADE1}\1\2019-05-24-18.mlp: Not connected.
What process generates these cache entries? We have no need for them and I’d like to stop the unnecessary busy-work that leads to their creation.
Here are the log entries surrounding the most recent system lockup:
2019-05-24 18:16:19 Playlist Start: \\mediator\media\SARAH-CLONE\March 26\04 Guts - 02-Mucagiami.wav
2019-05-24 18:20:02 Playlist Start: C:\mAirListData\sponsorship\carts\27186.wav
2019-05-24 18:20:06 Playlist Stop: \\mediator\media\SARAH-CLONE\March 26\04 Guts - 02-Mucagiami.wav (226 seconds)
2019-05-24 18:20:29 Error Lost connection to Database: [FireDAC][Phys][PG][libpq] :
2019-05-24 18:20:34 Playlist Stop: C:\mAirListData\sponsorship\carts\27186.wav (32 seconds)
2019-05-24 18:20:50 Error Error connecting to Database: [FireDAC][Phys][PG][libpq] could not connect to server: Connection timed out (0x0000274C/10060)
Is the server running on host "mairlist-db.rrr.internal" and accepting
TCP/IP connections on port 5432?
2019-05-24 18:20:52 Playlist Start: C:\mAirListData\sponsorship\carts\27184.wav
2019-05-24 18:20:52 Information RRRLOG: reset played spot links
2019-05-24 18:20:55 Information Connected to Database
2019-05-24 18:21:27 Playlist Stop: C:\mAirListData\sponsorship\carts\27184.wav (34 seconds)
At 18:16:19 mAirList was commanded to play a music file by a contact closure on the desk.
At about 18:02, the operator commenced a manual fade-out of the music and fired the cart that was cued in a different player.
That cart was commanded to play at 18:20:02. It was linked to a following cart.
At 18:20:28 I broke the connection between studio and core switch. Cart 27186 continued rendering. It stopped at the expected time (18:20:34).
The linked cart that followed it did not start. I then noticed the UI had frozen.
At 18:20:52 I reconnected the switches and Cart 27184 automatically commenced playing.
In this contrived scenario we were off air for 18 seconds and the operator was unable to interact with mAirList to remedy the situation.
What information can we collect / provide to help determine the cause of the lock up?
Cameron