Media Manager for Automotive Infotainment (Part 2)

By ICS Development Team Thursday, March 31, 2016

Part 2 Indexing and Tagging Media

This is the second in a series of blog posts about media manager components. In the first blog post we discussed the goals and a general architecture for our media manager. In today’s installation, we will show how to address some of the challenges when multiple users bring devices into the car.

Requirements for Media Indexing

As we discussed in the preceding blog post: When people drive in cars they usually have a phone, a tablet or some wearable device with them that provides access to media in some form. This could be on the device itself or streamed through some internet based service. In simple terms: Users want to listen to their music (or play videos). They do not care where it comes from. They also want to be able to find quickly what they are looking for and this results in some of the following challenges:

In a car we can have multiple users (driver/passengers)
Media comes from multiple devices from each of these passengers
Media is brought into the vehicle in a multitude of formats (many of which may not even be known at the time the car is manufactured)
Media comes and goes with passengers entering and leaving the car, with the car’s location or e.g,. the availability of network connections.

We find that at the heart of a solution to handle these challenges is the organization of media. Cataloging and indexing is the key to providing the needed functionality.

In this blog, we will show a simple cataloging and indexing example -- organizing media on a USB thumb drive -- and explain how it fits into our media manager architecture and addresses the challenges outlined above. While USB drives might not be ubiquitous, our example clearly shows how to deal with the problem in a general manner. The idea behind it can be extended easily to Media Players such as iPods and mobile phones.

On the challenge of indexing media across different sources

The problem of indexing media is two-fold: first, the files must be found, identified and the results stored. Second: media contained in files and streams must be classified, i.e., we are looking to answer questions like: How long is this “mp3” file? Who sang this song? Who directed this orchestra? All of this information should be available to the user as fast as the medium permits while preserving an always responsive, modern user experience.

The open source community, having long produced some outstanding quality multimedia software, has produced a few notable indexing solutions. The Gnome projects Tracker and KDE’s Nepomuk are powerful and complex search and indexing solutions appropriate for desktop solutions. Light Media Scanner (LMS), and FFMpeg project’s ffprobe are more suited for constrained environments and use cases.

Another widely used option is MediaInfo which is highly customizable, can easily be integrated in C++ based applications, has support for hundreds of media types and is a fast and robust solution with a long standing track record.

While MediaInfo itself provides a command line utility that can produce a wide variety of output formats we will integrate MediaInfo as a library into a Qt based application that will output metadata information in JSON format.

Instructions on how to build MediaInfoLib as a static library that we can link against can be found here and here and are not part of this article.

A complete listing and detailed instructions on how to build the code can be found on github at https://github.com/rkrauseics/QIcsMediaInfo .

Our example begins by including a few needed Qt classes for dealing with directories, text streams, JSON documents, objects, and arrays and parsing the command line.

MediaInfo’s API is mostly simple, however configuration through options is not entirely straightforward and the documentation is not of great help. MediaInfo::Option() can be used to configure MediaInfo’s behavior and for some options it returns strings. It can also be used to customize the output from parsing files by supplying strings with a certain “parameter” syntax.

E.g. to achieve an output of the form:

“AC/DC”|”Back in Black”|”Rock”

One would need to specify the parameters:

“General;%Performer%|%Album%|%Genre%\\n”

Parameters must be enclosed in % symbols. Note that the “\\n” is arbitrarily chosen to separate results for each file, while the “|” symbol is chosen to separate parameter results for individual files.

Listing 1 shows an easy way to configure the desired options and format the option string:

int main(int argc, char *argv[])
{
    QCoreApplication a(argc, argv);
    // Create Options for MediaInfo, there is a bunch more but these seem somewhat most relevant
    QStringList generalParams;
    generalParams << "CompleteName" << "FolderName" << "FileName" << "FileExtension"
                  << "FileSize" << "Duration" << "FrameRate" << "BitRate" << "BitRate_Mode/String"
                  << "Format" << "InternetMediaType"
                  << "Title" << "Season" << "Movie"
                  << "Album" << "Album_More" << "Album/Performer"
                  << "Part" << "Track" << "Track/Position" << "Compilation"
                  << "Performer" << "Genre" << "Mood"
                  << "Released_Date" << "Original/Released_Date" << "Recorded_Date"
                  << "Encoded_Application" << "Encoded_Library" << "BPM"
                  << "Cover" << "Cover_Mime" << "Lyrics"
                  << "Added_Date";
    QString generalInform=QStringLiteral("General;");
    foreach(QString s, generalParams) {
        generalInform += QString("\%%1\%|").arg(s);
    }
    generalInform+="\\n";

Listing 1: Qt Application and setup of “General Info Parameters” for MediaInfo

There are two classes with near identical APIs for working on either a single file or a directory of files respectively. MediaInfoLib provides recursive search of files under a given directory. However, there is no easy way to restrict the indexing to files with known endings. This might well lead to performance issues. Our example demonstrates the use of MediaInfoList which will work on directories of files but disables recursive behavior.

We instantiate an object of the class and then set parsing options (Listing 2)

    // ... 
    #define __T(__x) L ## __x
    // ...
    // Create a MediaInfoList object, this is for parsing entire directories
    MediaInfoList MI;
    MI.Option(__T("ParseSpeed"), __T("0")); 
    MI.Option(__T("Language"), __T("raw"));
    MI.Option(__T("ReadByHuman"), __T("0"));
    MI.Option(__T("Legacy"), __T("0"));

Listing 2: MediaInfoList and options to accelerate parsing speed

The code to process command line options is omitted here for brevity. Shown in Listing 3 is the input of the directory name and how the directory scan is started in non-recursive mode:

  QString dirPath;
  if(cmdLine.positionalArguments().count() > 0)
      dirPath = cmdLine.positionalArguments()[0];
  // ...
  QFile outFile;
  QTextStream outStream(stdout);
  // ...
  // This causes MediaInfo to open all files in the directory
  int nfiles = MI.Open(dirPath.toStdWString(),MediaInfoLib::FileOption_NoRecursive);
  qDebug() << Q_FUNC_INFO << "Opened " << nfiles << " files";

Listing 3: Opening the directory to parse with the “NoRecursive” option

The result of the scan is retrieved with the MediaInfo::Inform() function which returns a wchar variant of a std:basic_string. Note that the results for all files found are concatenated into the single string. The format parameter “\\n” that we appended (see Listing 1) will allow us to split the string:

  // Now we query MediaInfoLib for the data we are interested
  MI.Option(QStringLiteral("Inform").toStdWString(), generalInform.toStdWString());
  QString informOptionResult=QString::fromStdWString(MI.Inform());
  // Done - be good and close the MediaInfo object
  MI.Close();

Listing 4: Retrieval of results

The final part of the code uses Qt’s JSon implementation and basic QString functionality to quickly and safely split the results into QStrings for each parsed file and then further split each QString into results for each “General Parameter”. A QVariantMap matches each result with the corresponding “Parameter Name”. The QVariantMap is then converted into a JSON object and stored in a JSon array depending on file type. This wasn’t strictly necessary but eases further processing of the data. Listing 5 shows the process of converting and printing the resulting JSON document which concludes our small example:

  // Trivial bit of QJson<Magic> and output
  QJsonArray audioFiles;
  QJsonArray videoFiles;
  QStringList informResult=informOptionResult.split('\n',QString::SkipEmptyParts);
  if (informOptionResult.isEmpty()) {
      qWarning() << "could not find audio or video files in:" << dirPath;
      exit(0);
  }
  QVariantMap resMap;
  foreach (QString res, informResult) {
      QStringList resList=res.split("|");
      for (int i=0;i<resList.count()-1;++i) {
          resMap[generalParams[i]] = resList[i];
      }
      QJsonObject resObject=QJsonObject::fromVariantMap(resMap);
      QString mimeType=resMap["InternetMediaType"].toString();
      if (mimeType.startsWith("audio")) audioFiles.append(resObject);
      else if (mimeType.startsWith("video")) videoFiles.append(resObject);
      else {
          qWarning() << Q_FUNC_INFO << "mimetype for file"
          << resMap["CompleteName"]<< "not one of audio or video but"
          << resMap["InternetMediaType"];
      }
  }
  QJsonObject jsonObject;
  if (!audioFiles.isEmpty()) jsonObject.insert("audio",audioFiles);
  if (!videoFiles.isEmpty()) jsonObject.insert("video",videoFiles);
  QJsonDocument jsonDoc(jsonObject);
  outStream << qPrintable(jsonDoc.toJson());
}

Listing 5: Retrieving output from MediaInfo and converting it to JSON for output.

Remarks

Note that with our USB thumb drive example, we’ve glossed over some of the more complicated issues of connecting proprietary sources to the media manager. Different providers such as Apple, Google, Amazon and many others offer different hardware choices, interfaces and protocols to access available media. How can an automotive system stay current and relevant while at the same time being in the center of digital media activities?

We’ll cover ideas on how to deal with some of the issues in upcoming blog posts.

The source code for QIcsMediaInfo is available under the GPLv2 on github and contributions are warmly welcomed. The code is mostly intended to give a starting point for your own exploration of indexing media and in the next installment of our series, we will show how the idea was integrated in a plugin that supplies playlists to our media player.

Conclusion

We hope to have raised your interest in some of the detailed problems found when creating a Media Manager for an automotive or home media solution. Effective indexing of media files to achieve a consistent end-user experience is of great importance and the tools introduced and demonstrated today make the task solvable with moderate effort. Our next installment will put this code into the bigger architectural framework of the Media Manager and show how it can be used as a plugin component.

References:

Gnome Project, Tracker website, last accessed, March 31, 2016, https://wiki.gnome.org/Projects/Tracker
KDE Nepomuk user base wiki website, last accessed, March 31, 2016, https://userbase.kde.org/Nepomuk
Github open source website, Light Media Scanner, last accessed, March 31, 2016, https://github.com/profusion/lightmediascanner
FFMpeg Project, ffprobe website, last accessed, March 31, 2016, https://ffmpeg.org/ffprobe.html
Github MediaInfo website, last accessed, March 31, 2016, https://github.com/MediaArea/MediaInfo
Github ZenLib and MediaInfoLib websites, last accessed, March 31, 2016, https://github.com/rokrau/ZenLib/tree/master/Contrib & https://github.com/rokrau/MediaInfoLib/tree/master/Contrib
Github QIcsMediaInfo website, last accessed, March 31, 2016, https://github.com/rkrauseics/QIcsMediaInfo