One of the big (well, maybe it wasn’t all that big, but it was interesting) announcements of the 10.1 Flash Player was its ability to access detailed sound data from the microphone in the same way you can access mp3 data now. Until now, microphone access has been limited to volume. With the 10.1 player though, you can extract a bytearray of 512 sound samples just as you can from an mp3 file.
Generally speaking, the Microphone class, like the Sound class, now listens for the SampleDataEvent.SAMPLE_DATA event. When this is triggered, the event’s data property will hold a bytearray of 8192 samples. Those samples can then be “read into” a sound instance (similar to the way dynamic sounds are created) and the sound instance played. At that point, you can use the good ol’ SoundMixer.computeSpectrum() method to read the current sound samples into another bytearray and do with them as thou wilt.
Below is a quick example using a 3d grid of Papervision3D spheres. Just click to get started then speak/sing/whistle/holler into your mic (of course, you’ll need the 10.1 player installed and you’ll have to allow access to the microphone):
Seeing as how there doesn’t seem to be any documentation out there for doing this yet, for anyone interested, here’s a very basic example to play around with:
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 |
package { import flash.display.Shape; import flash.display.Sprite; import flash.events.Event; import flash.events.SampleDataEvent; import flash.media.Microphone; import flash.media.Sound; import flash.media.SoundMixer; import flash.utils.ByteArray; /** * Simple Microphone test for Flash Player 10.1 * @author Devon O. */ [SWF(width='550', height='450', backgroundColor='#000000', frameRate='40')] public class Basic extends Sprite { // sound private var _soundBytes:ByteArray = new ByteArray(); private var _micBytes:ByteArray; private var _micSound:Sound; private var _lines:Vector.<Shape> = new Vector.<Shape>(512, true); private var _ctr:int; public function Basic():void { if (stage) init(); else addEventListener(Event.ADDED_TO_STAGE, init); } private function init(event:Event = null):void { removeEventListener(Event.ADDED_TO_STAGE, init); initEqualizer(); initMic(); initSound(); addEventListener(Event.ENTER_FRAME, drawLines); } private function initEqualizer():void { var holder:Sprite = new Sprite(); for (var i:int = 0; i < 512; i++) { var line:Shape = new Shape(); with(line.graphics) { beginFill(0xFFFFFF); drawRect(0, -300, 1, 300); endFill(); } line.x = i; line.scaleY = 0; holder.addChild(line); _lines[i] = line; } holder.y = 400; holder.x = stage.stageWidth * .5 - holder.width * .5; addChild(holder); } private function initMic():void { var mic:Microphone = Microphone.getMicrophone(); if ( mic ) { mic.setLoopBack(false); mic.rate = 44; mic.gain = 60; mic.addEventListener(SampleDataEvent.SAMPLE_DATA, micSampleDataHandler); } else { // no mic } } private function micSampleDataHandler(event:SampleDataEvent) :void { _micBytes = event.data; _micSound.play(); } private function initSound():void { _micSound = new Sound(); _micSound.addEventListener(SampleDataEvent.SAMPLE_DATA, soundSampleDataHandler); } private function soundSampleDataHandler(event:SampleDataEvent):void { for (var i:int = 0; i < 8192 && _micBytes.bytesAvailable > 0; i++) { var sample:Number = _micBytes.readFloat(); event.data.writeFloat(sample); event.data.writeFloat(sample); } } private function drawLines(event:Event):void { SoundMixer.computeSpectrum(_soundBytes, true); if (_soundBytes.bytesAvailable) { _ctr = 0; while (++_ctr < 512) { _lines[_ctr].scaleY = _soundBytes.readFloat(); } } } } } |
EDIT:
Here is the above code compiled to .swf form, for those who’d like to check it out:
Can you please post the example as a swf? I can’t compile right now, but would like to see how it works. Thx
Can you tell me where I can download the flash sdk to compile this, I dont have the events.SampleDataEvent class.
I am using eclipse and flex builder 3.
Hey, Thor, I used the 3.5 Flex SDK and the new Global SWC, that you can download from the 10.1 player page: http://labs.adobe.com/technologies/flashplayer10/ (oh, and FlashDevelop :) )
Thanks for that, I forgot about the new Global SWC.
I am having another rather annoying problem now, everything is working except the eventlistner is just not being called
mic.addEventListener(SampleDataEvent.SAMPLE_DATA, micSampleDataHandler)
I have checked that the mic is not muted, and have tried with loopback both on and off, but it never gets called.
What am I doing wrong? can you post a single mxml file for me to try?
I am using flex builder 3 and eclipse 3 with flex sdk3.5
OK, as soon as you post something it works …
Sorry for the post, but I have just tried it on firefox, chrome and it works fine, but it doesnt work on IE8 on my development machine for some reason. It works on IE8 on a different machine when I post it on the web.
Glad you got it (somewhat) sorted. Very strange it doesn’t work in IE8 on one machine, but does on another. But I guess I’ve seen stranger things..
Thanks for the coffee!
d.
Does this work without playing back the sound that is coming in? I’d like to computeSpectrum of the mic input but I don’t want to have to playback and double up the sound.
Hey knuck, You can actually play around with the bytearray returned from the Microphone directly. In the example above you can just use the _micBytes bytearray instance in the soundSampleDataHandler() method and not write them into a new bytearray or play the sound. The drawback to this is that instead of 512 bytes (256 left/right) you get 8192 bytes and they are not nice clean normalized values.
I would like to capture a few seconds of voice and send it back to my web server in the WAV format. Is that possible? Do I need a WAV encoder? Any help is much appreciated!
Hey David, not really sure how you’d get the data back to your web server, but it shouldn’t be impossible.. There’s a nice article on the adobe developer site about recording sound snippets in AIR: http://www.adobe.com/devnet/air/flex/articles/using_mic_api.html And Didier Brun did an AIR to mp3 app using AIR’s new access to native processes and LAME: http://www.bytearray.org/?p=1142 Those would be the places I’d start anyway.
Good luck
Thank you, Devon. The first link seems to provide what I need. I’ll report back if I manage to get it work.
Cool. I might check it out this weekend if I have some time and see what I can come up with.
Just to update. I have done it with the help of the article below:
http://www.adobe.com/devnet/air/flex/articles/using_mic_api.html
which was kindly pointed out by Devon. Thanks!
I have a question. Does it require CS5 if you compile in the Flash IDE?
I installed the Flash Player 10.1 and I see it works in the browser. But when I compile it in Flash Pro CS4 it doesn’t work. Guess I need Flash Pro CS5?
Hey, Jim. The main thing is that you have to be able to compile to the 10.1 Flash Player. Obviously Flash Pro CS4 can’t do that. So you can use Flash Pro CS5 or you can use the Flex 3.5 or Flex 4.0 SDK (when I wrote this post, I was using the Flex 3.5 SDK and the FlashDevelop IDE).
Hi All,
I have a question I would like to develop a WEB Flash application which records the voice from user and saves it to user local file. I did it with Adobe Air but how I can transform it to WEB.
hey Vahe – see this post: http://blog.onebyonedesign.com/?p=531
Thanks a lot, now it works quite well :)
Computing the spectrum of the mic input.
Hi,
I notice that someone else has talked about computing the spectrum without playing it back, but I don’t quite understand the response.
How can you use the compute spec on the mic input without playing it back? Anything I try just gives zeros unless I actually play the sound back.
Thanks
Thor
Hey Thor, I don’t believe you can use the computeSpectrum method on the raw microphone bytes (though I may be wrong), but you can access them without playing the sound back by going straight into the bytearray holding the microphone data. In the above script, the bytearray _micBytes holds the raw microphone data. You can dig into that bytearray without first writing them into a _soundbytes array and playing that back. The _micBytes bytearray can get messy though, so be careful of that.
Hi,
Thanks for the reply, I should have made it a bit clearer.
I have done a lot of successful processing on the mic data, and have managed to access the byteArray etc.
The reason I want to use computeSpectrum is that it has a built in fft (spectrum calculator) that doesn’t seem to be anywhere else in flex. Of course I could write my own fft/spectrum function, but that would take time, and run much slower than the built in one. I want to use the computeSpectrum function if at all possible, but just playing the sound back to get the spectrum isn’t acceptable as the user of my software would get horribly confused as to why the speech is being played back to them at that time.
(It is speech recognition/language teaching software and sometimes you don’t want to play back what they said, just analyse it)
If adobe is smart enough to make a spectrum function, but stupid enough so it can’t be used sensibly then I will be a bit annoyed …
Is it not possible to just call computeSpectrum on given array, only the currently playing output sound?
It is very difficult writing speech recognition code in flex, and it runs very slowly, so anything to speed it up is very helpful.
Thanks
Thor
Hi Thor,
I’ve never seen anyone using computeSpectrum on a ‘non playing’ sound – and seeing as how it’s the SoundChannel rather than the Sound object itself that calls the method, I doubt that it is possible. Thinking off the top of my head, it may be somehow possible to load a bytearray into a sound instance, then use extract() to get cleaner data. Other than that, there may be something out there created in pixelbender or alchemy that could quickly and efficiently normalize a large bytearray. If I run across anything, I’ll post it, but don’t know of any way right this moment.
d.
I’ve made a Visualizer without playing the samples back.
Just operrated on the raw microphone output.
http://www.swfcabin.com/open/1282119401
Very cool!
nice post!
your example code doesn’t seem to work. would you be so nice to upload a .fla file with it? i’ld love to check it out.
If you don’t want the mic sound played back you can mute the sound playback with the SoundMixer. Just drop this in one of the init functions: SoundMixer.soundTransform = new SoundTransform(0);
I love this one! just what i needed!
Are you guys not getting crazy echoes and schreeching noises when testing any of the SWFs? The sound jumps back and forth between speaker and microphone and goes crazy :D OSX + Macbook Pro here.
I cant seem to get it to work when setting the SoundTransform to 0 like daniel suggested. No errors msgs, just no lines drawn
Same here. I can’t get how to access the raw data from the microphone neither (fp 11.2).
Actually managed to get the mic raw data:
package {
import flash.display.Shape;
import flash.display.Sprite;
import flash.events.Event;
import flash.events.SampleDataEvent;
import flash.media.Microphone;
import flash.media.Sound;
import flash.media.SoundMixer;
import flash.media.SoundTransform;
import flash.utils.ByteArray;
/**
* Simple Microphone test for Flash Player 10.1
* @author Devon O.
*/
[SWF(width=’550′, height=’450′, backgroundColor=’#000000′, frameRate=’40’)]
public class Basic extends Sprite {
// sound
protected static const LINES:int = 256;
private var _soundBytes:ByteArray = new ByteArray();
private var _micBytes:ByteArray;
private var _micSound:Sound;
private var _lines:Vector. = new Vector.(512, true);
private var _ctr:int;
public function Basic():void
{
if (stage) init();
else addEventListener(Event.ADDED_TO_STAGE, init);
}
private function init(event:Event = null):void
{
initEqualizer();
init();
}
protected function init():void
{
_soundBytes = new ByteArray();
var mic:Microphone = Microphone.getMicrophone();
mic.setSilenceLevel(0, 4000);
mic.addEventListener(SampleDataEvent.SAMPLE_DATA, micSampleDataHandler);
function micSampleDataHandler(event:SampleDataEvent):void
{
var i:int = 0;
while(event.data.bytesAvailable)
{
if(i >= LINES-1)
{
i = 0;
return;
}
//var sample:Number = event.data.readFloat();
//_soundBytes.writeFloat(sample);
_lines[i].scaleY = event.data.readFloat() * 3;
i++;
}
}
}
private function initEqualizer():void
{
var holder:Sprite = new Sprite();
for (var i:int = 0; i < LINES; i++)
{
var line:Shape = new Shape();
with(line.graphics)
{
beginFill(0xFFFFFF);
drawRect(0, -400, 400/LINES, 400);
endFill();
}
line.x = i * 400/LINES;
line.scaleY = 0;
holder.addChild(line);
_lines[i] = line;
}
holder.y = 200;
holder.x = stage.stageWidth * .5 – holder.width * .5;
addChild(holder);
}
}
}
Is there any way to get the actual frequency in numbers?
Interesting.. I’m not really sure if it makes sense to talk about frequency when dealing with voice. But I’m definitely no sound expert. Best info (well at least as a starting point) I could find is this stack overflow question: http://stackoverflow.com/questions/2361895/how-to-determine-the-frequency-of-the-input-recorded-voice-in-iphone
I’m trying to make a guitar tuner type of application. For that I need to know the exact frequency number in Hz. I tried a lot to find a good answer but no luck. Basically, I’m trying to convert that ByteArray info into a frequency number.