The researchers have developed a production-ready platform that can control mobile devices, PCs, and the Internet of Things (IoT) devices, enabling a bring-your-own-device (BYOD) experience of music-driven multimedia performance via the Internet.

“Songle Sync” a platform that can control hundreds of a variety of devices synchronized with music playback

How to use Songle Sync

Music videos with synchronized sound and video have become commonplace, and a variety of visual effects synchronized with music playback in concert venues are now a frequent sight. However, such music-driven multimedia performances conventionally require dedicated devices or applications that were developed individually, resulting in poor expandability and high costs. Because of this, there has been strong demand to develop technology that enables easy and inexpensive development of music-driven applications, using a lot of various devices connected to the Internet.

The researchers developed a large-scale web-based platform named “Songle Sync ( https://api.songle.jp/sync ),” which enables programmers to control hundreds of a variety of devices in synchronization with music playback. Using the results of analysis with the previously developed “Songle ( https://songle.jp ),” the researchers developed a technology that can drive multimedia effects in time with music playback by synchronizing smartphones, PCs, and other Internet-connected devices via intermittent compact communications with a Songle Sync server. In addition, they implemented the platform on top of web standard technologies so that a variety of devices can be controlled in a unified way, thereby improving scalability and reducing cost.

AIST plans to use this “Internet of Musical Things (IoMT)” technology that controls many mobile devices, PCs, and IoT devices simultaneously in various scenes including live events, shopping malls, and city streets, in cooperation with industry.