Realtime lip-sync code snippet for Flash using AS3

August 17, 2011 Flash, Programming, Tips & Tricks

One of the flash projects I have been working on has required a lip sync talking. Dialogues were dynamic so frame by frame matching was out of the question. So, I ended up with the following action script 3 code which I’m sharing with you here…

It processes the input sound in real time, which returns the different values based on the sound volume and also outputs the sound itself.

You can use those values to match up mouth animation frames as provided in the sample. But you may also simulate the effect using a single mouth image and scaling it based on the values returned by the function…

Usage

FLA project is compatible with Flash CS4, contains sample speech & mouth animation images.
speech() is the the sound file linked from the library.
mouth is the animation clip that contains mouth drawings.

Downloads

Sample SWF file for preview.
Download FLA project

License

This code is free to use, distribute, modify and study. Mouth animation and the images provided in the sample FLA is NOT free to use in anyway. When referencing please link back to this website / post in any way e.g. direct link, credits etc. If you find this useful, please leave a comment and share using the buttons below!

/*
- Copyright 2011 Pixel Envision (E.Gonenc)
- http://www.pixelenvision.com/
- support@pixelenvision.com
*/

var left: Number;

function processSound(event: SampleDataEvent): void {
  var bytes: ByteArray = new ByteArray();
  playerObject.sourceSnd.extract(bytes, 4096);
  bytes.position = 0;
  while (bytes.bytesAvailable & gt; 0) {
    left = bytes.readFloat() * 128;
    if (left & lt; 0) {
      left = -left;
    }
    var scale: Number = left * 2;
  }
  event.data.writeBytes(bytes);

  //Define mouth animation here
  if (scale & lt; 1) {
    mouth.gotoAndStop(1);
  } else if (scale & lt; 10) {
    mouth.gotoAndStop(2);
  } else if (scale & lt; 25) {
    mouth.gotoAndStop(3);
  } else if (scale & lt; 50) {
    mouth.gotoAndStop(4);
  } else {
    mouth.gotoAndStop(5);
  }
  //Define mouth animation here

  trace(scale);
}

var playerObject: Object = new Object();
playerObject.sourceSnd = new speech();
playerObject.outputSnd = new Sound();
playerObject.outputSnd.addEventListener(SampleDataEvent.SAMPLE_DATA, processSound);
playerObject.outputSnd.play();
stop();

- http://www.pixelenvision.com/

- support@pixelenvision.com

var left: Number;

function processSound(event: SampleDataEvent): void {

var bytes: ByteArray = new ByteArray();

playerObject.sourceSnd.extract(bytes, 4096);

bytes.position = 0;

while (bytes.bytesAvailable & gt; 0) {

left = bytes.readFloat() * 128;

if (left & lt; 0) {

left = -left;

}

var scale: Number = left * 2;

}

event.data.writeBytes(bytes);

//Define mouth animation here

if (scale & lt; 1) {

mouth.gotoAndStop(1);

} else if (scale & lt; 10) {

mouth.gotoAndStop(2);

} else if (scale & lt; 25) {

mouth.gotoAndStop(3);

} else if (scale & lt; 50) {

mouth.gotoAndStop(4);

} else {

mouth.gotoAndStop(5);

}

//Define mouth animation here

trace(scale);

}

var playerObject: Object = new Object();

playerObject.sourceSnd = new speech();

playerObject.outputSnd = new Sound();

playerObject.outputSnd.addEventListener(SampleDataEvent.SAMPLE_DATA, processSound);

playerObject.outputSnd.play();

stop();

6 Responses to Realtime lip-sync code snippet for Flash using AS3

richard

1 / 17 / 2013

hey..can i ask??why is it that u use 4096??

richard 1 / 17 / 2013

Erdener

1 / 17 / 2013

Since the lip-sync is done with streaming sound, 4096 looked like a good value to work with considering memory / lip-sync frame rate.

Erdener 1 / 17 / 2013

Erik Hermansen

12 / 6 / 2012

One more thing… if you change the line under “Define Mouth Animation” to the one below, it will close the mouth at the end of audio more reliably.

if (scale<1 || isNaN(scale)) {mouth.gotoAndStop(1);}

Erik Hermansen 12 / 6 / 2012
Erik Hermansen

12 / 6 / 2012

I used this to get my own lip sync animation to work. Thank you!

Erik Hermansen 12 / 6 / 2012

Erdener

12 / 6 / 2012

You’re welcome, I’m glad it’s been useful for you…

P.S. I’ve finally took few seconds and changed my name on posts, thanks to you… :)

Erdener 12 / 6 / 2012