Media Manager for Automotive Infotainment (Part 4)
Media Manager Architecture - From Indexing Media to Playing
In the first two parts of our blog series, we examined some of the functional requirements of our Media Manager. We also showed a solution for indexing of media. In today’s installment, we explain the core architecture of the media manager and how indexed media information can be presented to our end users.
To summarize, we want the Media Manager to have a few important capabilities:
-
Present media from different sources, e.g., Phones, Tablets, Internet Streaming Source and USB sticks, to the user in a unified way that allows for easy sorting, searching and filtering.
-
Play the media in dynamically created playlists using the methods mentioned above.
-
Recognize insertion or removal of media sources without user interaction and dynamically update the playlists.
-
Persistent storage of state, playlists, media sources etc..
-
Allow for a modern and appealing user interface (UI) to select media playlists, control the media players and display information about available media as well as to browse, search, sort and filter available media.
-
Allow for phone and tablet-based interfaces to do the same as above plus to allow additional functionality e.g. the advanced creation of playlists.
This is not a complete list of all possibilities that one might want to incorporate but it does contain the most important ones. The Media Manager should allow for a user experience that integrates all brought in as well as car-specific infotainment options. It must be intelligent, extensible and most importantly it must allow users to enjoy their media in the most simple way possible.
From a software architecture point-of-view, extensibility is nearly always achieved by building a small and functionality-restricted core. Then we must decide how to enrich this core with interfaces that we can implement in plugins. Handling of Plugins using Qt is especially straighforward - the definition of interfaces is easy - there is no need for IDLs and complicated code generation tools. We will desribe the details of the Media Manager plugins in out next blog but first let's move on and see what the core idea of the Media Manager is. The following picture illustrates the data flow from the connected devices to media playlists.
Figure 1 - Dataflow of the Media Manager
Core Functionality of the Media Manager
The core functionality of our Media Manager consists of taking media data from the devices that store it and presenting this data to media players that can play it. It is important to implement this without copying any of the actual media data.
For this purpose we first introduce a data structure to store playlist information indexed from a device, let’s call this Media Source. The corresponding C++ class is class MediaSource (Listing 3). For each uniquely identified device, a MediaSource object is instantiated and stored.
#ifndef MEDIASOURCE_H
#define MEDIASOURCE_H
#include <QObject>
#include <QUrl>
#include <QJsonObject>
class MediaDeviceInterface;
/*!
* \brief MediaSource
* This class caches source lists from devices.
* in JSonObjects that contain the DeviceUrl
* a JsonArray with AudioFiles,
* a JSonArray with VideoFiles
*
*/
class MediaSource : public QObject
{
Q_OBJECT
Q_PROPERTY(QJsonObject mediaSourcePlaylist READ mediaSourcePlaylist
WRITE setMediaSourcePlaylist NOTIFY mediaSourcePlaylistChanged)
Q_PROPERTY(QUrl deviceUrl READ deviceUrl WRITE setDeviceUrl NOTIFY deviceUrlChanged)
public:
/** Creates a MediaSource from a device, a Url and a parent.
* The MediaSource will become the parent of the device.
**/
explicit MediaSource(MediaDeviceInterface * device, const QUrl & deviceUrl,
QObject * parent = 0 );
void updateMediaSourcePlaylist() ;
QJsonObject mediaSourcePlaylist() const;
/** Check whether the MediaType is present in the MediaSourcePlaylist */
bool hasMediaType(const QString & mediaTypeStr) const;
/** Returns a QJsonArray for the given MediaType.
* The returned QJsonArray will be empty if the MediaType was not present
**/
const QJsonArray mediaArray(const QString & mediaTypeStr) const;
/** Returns the deviceUrl that is associated with this MediaSource as a string */
const QString deviceUrlString() const;
/** Returns the deviceUrl as QUrl */
const QUrl deviceUrl() const;
signals:
void mediaSourcePlaylistChanged(const MediaSource * mediaSource);
void deviceUrlChanged(QUrl deviceUrl);
private slots:
void setDeviceUrl(QUrl deviceUrl);
void setMediaSourcePlaylist(QJsonObject mediaSourcePlaylist);
private:
MediaDeviceInterface * m_device;
QJsonObject m_mediaSourcePlaylist;
QUrl m_deviceUrl;
};
#endif // MEDIASOURCE_H
Listing 3: MediaSource
This object contains a JSON Object consisting of multiple JSON Arrays, one per MediaType present on the device. Each JSON Array is an array that contains indexing data, again JSON Objects, one for each media item. These JSON Objects contain attributes of a single media item such as e.g., file names, artists, cover art and many other things of interest to the end user.
We will call the JSon Array that contains the objects for each item MediaPlaylist. Assume that we retrieved one or more MediaPlaylists from the Media Indexer as described in Part 2 of our Blog (Part 2:Listing 5). They are made of items of the same MediaType. Examples for these MediaTypes are “audio files”, “video files”, “Bluetooth audio streams”, “audio DLNA”, “avb ethernet video” and much more. Important here is that a MediaType identifies a data format that specific MediaPlayers are capable of rendering.
{
"AudioFileMediaType": [
{
"Album": "Southernality",
"Artist": "A Thousand Horses",
"CompleteName": "/mm_test/audio/a.mp3",
"Title": "(This Ain’t No) Drunk Dial",
},
{
"Album": "Billboard Top 60 Country Songs",
"Artist": "Big & Rich",
"CompleteName": "/mm_test/audio/b.mp3",
"Title": "Run Away with You",
}
],
"VideoFileMediaType": [
{
"CompleteName": "/mm_test/video/mad_max.mp4",
"FileName": "mad_max",
"Format": "MPEG-4",
"InternetMediaType": "video/mp4",
},
{
"CompleteName": "/mm_test/video/sup-vs-bat.mp4",
"FileName": "sup-vs-bat",
"Format": "MPEG-4",
"InternetMediaType": "video/mp4",
}
]
}
Data Structure: JSON Object containing Audio and Video file playlists
The second data structure we introduce is the MediaSession which holds sets of MediaPlaylists (the JSONArrays mentioned above). The corresponding C++ class is class MediaSession and shown in Listing 4. In addition to the contributing playlists, a MediaSession object also has a reference to a MediaPlayer that is capable of playing media of the corresponding MediaType.
#ifndef MEDIASESSION_H
#define MEDIASESSION_H
#include <QMap>
#include <QObject>
#include <QJsonArray>
class MediaPlayerInterface;
class MediaSession : public QObject
{
Q_OBJECT
public:
explicit MediaSession(MediaPlayerInterface * player, QObject *parent = 0);
/** Append a playlist from a MediaSource to this MediaSession
* This function keeps track of the playlists coming from the deviceUrl.
*/
void appendMediaSourcePlaylist(const QString deviceUrl, const QJsonArray playlist);
/** Remove the playlist from deviceUrl and rebuilt the playlist for the MediaPlayer
*/
void removeMediaSourcePlaylist(const QString deviceUrl);
/** Access to the MediaPlayer */
const MediaPlayerInterface * player() const { return m_player;}
/** This function returns the concatenated playlist with all the meta-information as
* a QJsonArray. This list is not cached to calling this function will sort, filter
* etc and return the full result of this effort.
**/
const QJsonArray mediaSessionPlaylist() const ;
signals:
void mediaSessionPlaylistChanged(QStringList mediaSessionPlaylist);
private slots:
private:
/** This function rebuild the MediaSession playlist that is then sent to the player
* immediately, the list is not stored, if you need it then get it from the player.
**/
void rebuildMediaSessionPlaylist();
/** MediaPlayer
**/
MediaPlayerInterface * m_player;
/** Cache all contributing source playlists.
* Used to rebuild the session playlist fast after adding and removing.
**/
QMap<QString,QJsonArray> m_sourcePlaylists;
};
#endif // MEDIASESSION_H
Listing 4: MediaSession
An essential function of our Media Manager is now the following:
When a device is connected, e.g. a USB pen drive is plugged in, the Media Manager receives a notification from the Device Manager plugin. With the help of a suitable MediaDevice indexing results in a MediaSource object that is delivered to and received by the Media Manager. The Media Manager stores and accesses MediaSession objects corresponding to MediaTypes contained in the MediaSource, creating new ones if necessary. It then appends the playlists coming from the MediaSource object to the MediaSession.
During this step, filtering and sorting can be applied before the result is delivered to the MediaPlayer referenced by the MediaSession. Again, MediaSessions store sets of playlists (in JSON Arrays) identified with the MediaSource they came from, hence, it is trivial to update playlists upon removal of a device at the cost of rebuilding the playlist and transferring it to the MediaPlayer again. Here it should be noted that the use of “implicitly shared” container classes is fundamental to a robust and efficient implementation. Listing 6 contains the Media Manager function that updates the MediaSession while Listing 7 contains the MediaSession function that rebuilds the playlist from all current MediaSources of a MediaSession and hands the result to the MediaPlayer.
void MediaManager::updateMediaSession(const MediaSource * mediaSource)
{
const QString deviceUrlStr=mediaSource->deviceUrlString();
for (int mt=(int)NoMediaType+1;mt<(int)EndMediaType;++mt)
{
MediaType mediaType=(const MediaType)mt;
QString mediaTypeStr=mediaTypeToString(mediaType);
const QJsonArray playListArray=mediaSource->mediaArray(mediaTypeStr);
if (playListArray.isEmpty()) {
continue;
}
MediaSession * mediaSession=0;
if (mediaSessions.contains(mediaType)) {
mediaSession=mediaSessions[mediaType];
}
else {
MediaPlayerInterface * player=0;
if (mediaPlayerPlugins.contains(mediaType)) {
player=mediaPlayerPlugins[mediaType];
}
else {
qWarning() << Q_FUNC_INFO << "no plugin found for MediaType"
<< mediaType;
continue;
}
mediaSession=new MediaSession(player, this);
mediaSessions.insert(mediaType, mediaSession);
}
mediaSession->appendMediaSourcePlaylist(deviceUrlStr,playListArray);
// the first one active until the user changes it through the controller
if (mediaSessions.count()==1)
setActiveMediaSession(mediaType);
}
}
Listing 6: Update of the MediaSession
void MediaSession::rebuildMediaSessionPlaylist()
{
QStringList playList;
foreach (const QJsonArray jsa, m_sourcePlaylists) {
foreach (QJsonValue jv, jsa) {
QJsonObject jo=jv.toObject();
const QString & f=jo["CompleteName"].toString();
playList << QString("file:%1").arg(f);
}
}
emit mediaSessionPlaylistChanged(playList);
m_player->setMediaPlaylist(playList);
qDebug() << Q_FUNC_INFO << playList;
}
Listing 7: Rebuild Media Session Playlist
It is up to the MediaPlayer implementation to decide what to do when they receive an updated playlist. A smart implementation of a MediaPlayer might be able to replace its playlist upon receiving a current one without interrupting the current playing song if it is contained in both playlists. We will show such an implementation in an upcoming Blog.
Our Media Manager employs the concept of active MediaSessions to control the actual playback of media. Hence, it calls the active sessions player with the standard actions of playing media e.g. play, pause, next, previous, play by index etc.. The control of the Media Manager itself is through a MediaManagerControllerInterface that is implemented by a variety of “stateless” plugins. E.g., a simple UI plugin allows for a graphical user interface to be implemented while a “remote controller” plugin allows mobile devices to control the Media Manager and thus its playing functionality. We will describe the functioning and implementation of these plugins in our next blog installments so stay tuned and in the meantime fork our code on GitHub and give it a spin.
Conclusion
We have shown and demonstrated the main idea behind our Media Manager implementation: Using Plugins and abstract Interfaces to move, filter and sort media indexing data from devices to players. In our next installment, we look at how the process can be controlled by the end user through a wide variety of user interfaces from in-car center stack HMIs as well as mobile applications.