{"id":5297,"date":"2022-04-26T12:04:51","date_gmt":"2022-04-26T12:04:51","guid":{"rendered":"http:\/\/procedural-audio.com\/blog\/?p=5297"},"modified":"2022-04-27T01:16:24","modified_gmt":"2022-04-27T01:16:24","slug":"live-2d-and-gamesynth","status":"publish","type":"post","link":"http:\/\/procedural-audio.com\/blog\/2022\/04\/26\/live-2d-and-gamesynth\/","title":{"rendered":"Live2D and GameSynth"},"content":{"rendered":"

Live2D Cubism<\/a> is a popular animation software among virtual YouTubers and game developers. It allows for the creation of dynamic expressions and anime-style animations, simply by applying deformations to 2D illustrations.<\/p>\n

GameSynth 2022 now supports the import of Live2D animation curves, in order to design sounds that perfectly match these animations.<\/p>\n

\"Live2D\"
\n<\/a><\/p>\n

Importing Animation Curves<\/h2>\n

Live2D\u2019s animation curves can be exported via the File->Export for runtime-> Export motion file<\/em> command. They are stored in motion3.json <\/em>files that can be found in the runtime>motions <\/em>subfolder of a Live2D project.<\/p>\n

To convert them to Automation Curves<\/strong> usable in GameSynth, click on the Automation Curves menu, select Import curve<\/em>, and choose a motion3.json file. Then, select a curve in the window that lists all the parts of your character (Eyes, Arms, Mouth, Legs, etc..). It is possible to visualize the curve and process it (Normalization, Clipping, Thinning) before importing it. Note that the Duration is also retrieved from the motion file.<\/p>\n

\"Live2D\"
\n<\/a><\/p>\n

In this example, the Nito<\/a> sample character provided by Live2D was used. As shown above, the animation in Live2D matches exactly the automation in GameSynth.<\/p>\n

Let\u2019s see how to use these curves to design anime-like sound effects for this cute character.<\/p>\n

Two curves were imported, one for character\u2019s mouth opening, and the other for the body\u2019s quick back and forth.<\/p>\n

The Body curve already looks like an LFO, which is great to control both the Amplitude<\/em> and Pitch<\/em> of a Sine Bank<\/strong> and create a feeling of dizziness. A Mapper<\/strong> between the Automation Curve<\/strong> and the Sine Bank<\/strong> controls the range of each parameter. An additional Chorus<\/strong> module emphasizes the effect by adding even more modulation.<\/p>\n

Another branch of the patch uses the Mouth Automation<\/strong>. This time, the curve is simpler, only consisting of the mouth opening, holding, and closing. It will be used to control the amplitude<\/em> of a Glottis<\/strong> module, ideal to create vocalization. The Pitch<\/em> is also controlled, after reducing its range with a Mapper<\/strong> and creating random modulations to simulate the hoarseness of the voice with a Drifter<\/strong> module. Finally, a Formant Filter<\/strong> softens the signal to produce a more natural sound.<\/p>\n

\"Live2D\"
\n<\/a><\/p>\n

Let\u2019s increase the complexity a bit: using a different animation for the same character, four curves are imported, for the Eye, Mouth, Arm and Body.<\/p>\n

The Eye and Mouth curves are used in the same way, although we adjust the Mapper<\/strong> parameters to provide a different pitch range, to convey a different emotion. As for the Eye, a Derivative<\/strong> module to gets the Velocity<\/em> of the curve and triggers the sound only when the Eyes are closing then opening.<\/p>\n

The same technique is used for the Arms curve, where a Derivative<\/strong> module controls the Amplitude<\/em> and Pitch<\/em> of a Noise Bands<\/strong> module to generate cloth Foley corresponding to the arms movement.<\/p>\n

Finally, the straight curve of the Body controls both the Amplitude<\/em> and Pitch<\/em> of another Sine Bank<\/strong> module, creating a sweep sound that matches the jumping movement of the character.<\/p>\n

\"Live2D\"
\n<\/a><\/p>\n

Going further<\/h2>\n

Importing an animation curve in a sound tool offers many benefits. Not only it\u2019s possible to synchronize the sound perfectly with the animation, but should the duration or movement change during production, a new sound can be generated instantly, simply by reimporting the new curve. This is especially useful in highly iterative production environments such as game development. It eliminates the need to source new sounds or to process existing ones.<\/p>\n

To demonstrate this, let\u2019s design sounds for the Rice Glassfield<\/a> sample character, also provided by Live2D, which is a bit more game-oriented.<\/p>\n

The right arm, hand light, and magic power curves will be used. For the first two, the patching is similar to the previous examples. Cloth movements sound are based on a Noise Bands<\/strong> module for the right arm, and shimmery, high-pitched sounds are generated for the hand light using Sine Bank<\/strong> and Oscillator<\/strong> modules.<\/p>\n

In the case of the Magic Power, the goal is to trigger a sound synchronized with the graphic effect on screen. To achieve this, a Crossings<\/strong> module is used. It will activate its output when its input reaches a specific value while the curve is increasing, decreasing, or in both cases. Since the curve has an impulse shape, a value of 0.03 when the curves increases will work well here. The Trigger mode<\/em> is set to Always, <\/em>in case the magical power is triggered more than once (as we will see below). Sound-wise, a Chaos<\/strong> module generates the noisy part of the sound, while a Sine Bank<\/strong> module synthesizes the shimmery part.<\/p>\n

\"Live2D\"
\n<\/a><\/p>\n

During production, an animation might be updated, for example here to go faster, or use the Magic Power twice. Once the edits made in Live2D, the only task is to re-import the animation curves from the Automation window in GameSynth. In this example, the duration will be automatically adjusted, as well as the curves. Pressing Play in GameSynth will generate a new sound, matching perfectly the updated animation, without having to edit the patch!<\/p>\n

You can download the GameSynth patches used in this tutorial below, and the sample characters on Live2D download page<\/a> to try yourself!<\/p>\n

\"Live2D\"
\n<\/a><\/p>\n

 
\nPlease make sure you have the very latest GameSynth update<\/a>, as the Live2D importer in GameSynth 2022.1 has recently been improved.<\/span><\/p>\n

\"download-patches\"<\/a><\/p>\n

\"GameSynth\"<\/a><\/p>\n

\"Live2D\"<\/a><\/p>\n

Copyright Disclaimer: this content uses sample models owned and copyrighted by Live2D Inc. The sample models are utilized in accordance with terms and conditions set by Live2D Inc. This content itself is created at the author\u2019s sole discretion.<\/em><\/p>\n

 <\/p>\n

 <\/p>\n","protected":false},"excerpt":{"rendered":"

Live2D Cubism is a popular animation software among virtual YouTubers and game developers. It allows for the creation of dynamic expressions and anime-style animations,<\/p>\n","protected":false},"author":1,"featured_media":5323,"comment_status":"closed","ping_status":"closed","sticky":false,"template":"","format":"standard","meta":[],"categories":[6,15,12],"tags":[],"_links":{"self":[{"href":"http:\/\/procedural-audio.com\/blog\/wp-json\/wp\/v2\/posts\/5297"}],"collection":[{"href":"http:\/\/procedural-audio.com\/blog\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"http:\/\/procedural-audio.com\/blog\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"http:\/\/procedural-audio.com\/blog\/wp-json\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"http:\/\/procedural-audio.com\/blog\/wp-json\/wp\/v2\/comments?post=5297"}],"version-history":[{"count":22,"href":"http:\/\/procedural-audio.com\/blog\/wp-json\/wp\/v2\/posts\/5297\/revisions"}],"predecessor-version":[{"id":5321,"href":"http:\/\/procedural-audio.com\/blog\/wp-json\/wp\/v2\/posts\/5297\/revisions\/5321"}],"wp:featuredmedia":[{"embeddable":true,"href":"http:\/\/procedural-audio.com\/blog\/wp-json\/wp\/v2\/media\/5323"}],"wp:attachment":[{"href":"http:\/\/procedural-audio.com\/blog\/wp-json\/wp\/v2\/media?parent=5297"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"http:\/\/procedural-audio.com\/blog\/wp-json\/wp\/v2\/categories?post=5297"},{"taxonomy":"post_tag","embeddable":true,"href":"http:\/\/procedural-audio.com\/blog\/wp-json\/wp\/v2\/tags?post=5297"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}