Niko Helle

Flash is still the best tool for online video and there are more and more gorgeous videos where external media like images from user’s Facebook profile are embedded to a video. World famous examples are Tackfilm, Take this lollipop and Lost in val Sinestra.

I participated in a similar campaign and our technical partner had a lot of problems with video sync. How can you tell with 100% reliability which video frame is currently shown? Zeh Fernando writes about possible solutions in this  article. The solutions (and their cons) are:

– embedding the video inside a SWF and getting the current frame of that movieclip (very inconvenient and cumbersome)
– use Netstream’s time property (very inaccurate)
– use NetStream’s decodedFrames + droppedFrames (works until Flash player hangs for one frame or the video is restarted or rewinded)
– check cue points (requires old video codec)

Eric Decker came up with a better solution. Ingenious!  Embed the the frame number to each frame with a barcode. Sounds very tedious, but quite easy task once the ExtendScript and ActionScript are ready. I didn’t find good examples, so I had to make one.

Frame numbers cannot be plain text, because decoding shapes would be too heavy for the CPU (nevermind the coder) and would not be 100% reliable. So the frame numbers are embedded as a binary number, ergo ones and zeros. My solution uses pixels of certain color, because it is very easy to get the color value of a pixel with ActionScript. If the color value exceeds a threshod value, the pixel equals ”1”.  First I tried a simple 1px high row of pixels below  the video, but compression causes pixels to blend with adjacent pixels so that is not even 70% reliable. So the blocks presenting binary value must be larger than 1*1px. In my tests even 3*3px blocks weren’t enough, if the compression is set to medium. 5×5 pixels with 7px gaps between blocks work with 100% reliability. I bet smaller gaps work too, but I didn’t test it.

Compression, and the blending caused by it, also fades the color, so the 100% color value (like 255,0,0 for pure red) cannot be used either. That’s why the AS3 script has a threshold parameter for the color value. If the parameter is 50, colors values decoded as ”1” would be 127 (255*50%) and above. You can easily test which threshold value is the best.

Here’s a screenshot of the video with some explanation:

The After Effects ExtendScript

I’ve never used After Effects before, but the script is so simple there probably isn’t a simpler way to add a barcode to a video. The script increases the height of the composition by certain amount (block height +bitOffsetFromBottom +bitOffsetFromTop) and adds a solid colored bar in that position. Then the number of frames is calculated from the frame rate and length of the video. That number is converted to binary (eg. 10010011) and one shape is added to the bar for each needed number (8 in this example).

The script adds an expression to each shape which controls the opacity of each shape as the playhead moves. Each shape is linked to a position in the binary number and if the number in that position is ”0”  the opacity is 0 and vice versa.

The AS3 script

After the AE script is run, it shows an alert box with used parameters and  those should be copied to Flash. This way the parameters between AE and AS3 match and you don’t have to set them by hand. Parameters are:

bitWidth: width of the shape
bitHeight: height of the shape
bitSpacing: : spacing between shapes
bgColor: background color
bitColor: shape color, this value is 0, 1 or 2.  0 indicates pure red, 1 pure green and 2 pure blue
bitOffsetFromBottom: shape distance from the bottom
bitOffsetFromTop: shape distance from the top of the background bar

AS3 has also the threshold parameter mentioned above.

It is also quite easy to test if the barcode works. The AS3 script warns if frames are skipped or the calculated frame number is smaller that previous, indicating that the barcode was calculated incorrectly. This is caused by too small shapes for the compression level and the solution is to increase the shape size and/or spacing.

One problem: the height of the video increases and we don’t want to show the binary shapes to the user. The AS3 script includes three solutions for this: mask the video, draw a shape over the barcode or capture each frame as a bitmap and cut off the barcode. Of course you can just make the height of the HTML element smaller and disable scaling.

Here is a demo. Note that I’ve added a text field to the video that displays the current frame as a binary. The AS3 script adds a textfield too that show the binary calculated from the shapes. This is just for debugging purposes and those can be disabled in AE and Flash.

Demo

Source files and downloads

Here’s the AE script:

{
// FrameNumberToBinary.jsx
	
    // copyright Niko Helle, nikohelle.net

    
    var bitWidth = 5;   // width of the shape representing binary
    var bitHeight = 8; // height of the shape representing binary
    var bitSpacing = 7; // spacing between shapes
    var bitBGHeight = 0; // do not set, calculated below
    var bitColor = [1,0,0]; /// color of the shape [R,G,B] Use only full color values: [1,0,0] or [0,1,0] or [0,0,1]. Other values will not work.
    var fillColor = [0,0,0]; /// color of the bar [R,G,B]
     var bitOffsetFromTop = 3; // shape distance from the top of the bar
    var bitOffsetFromBottom = 3;  // shape distance from the bottom of the bar
   
	
	function FrameNumbering(thisObj)
	{
        
        //calculate height of the bar
        bitBGHeight = bitOffsetFromTop+bitHeight+bitOffsetFromBottom; 
     
        //get the composition
        var firstComp = app.project.item(1);

        //increase the height by the bar height
        firstComp.height = firstComp.height+bitBGHeight; 

        // check if the there is enough space for the binary numbers. If shapes and spacing are wide, the numbers may not fit the video 
        if(!checkFrames(firstComp)){ 
            return;
        };

        //create the bar
       var bar = firstComp.layers.addSolid(fillColor,"bitBG", firstComp.width, bitBGHeight, 1) 
     
        // reset anchor for easier positioning
       resetAnchor(bar); 
       
       //position the bar to the bottom of the composition
       setPosition(bar,0,firstComp.height-bitBGHeight,false); 
   
        //create shapes
       calcBits(firstComp,bitColor); 
       
        // determine which color is used, and alert it for the AS3, 0=R, 1=G, 2=B. Only solid, full colors can be used!
        if(bitColor[0] == 1) bitColor = 0; 
        else if(bitColor[1] == 1) bitColor = 1;
        else bitColor = 2;       
       
       //alert used parameters so they can be copied to AS3
       
       var alertTxt = "Copy these values to the ActionScript file: \n";
       alertTxt +="_bitWidth="+bitWidth+";\n";
       alertTxt +="_bitHeight="+bitHeight+";\n";
       alertTxt +="_bitSpacing="+bitSpacing+";\n";
       alertTxt +="_bgHeight="+bitBGHeight+";\n";
       alertTxt +="_bitColor="+bitColor+";\n";
       alertTxt +="_bitOffsetFromBottom="+bitOffsetFromBottom+";\n";
       alertTxt +="_bitOffsetFromTop="+bitOffsetFromTop+";\n";
       alertTxt +="_frames="+(firstComp.frameRate*firstComp.duration)+";\n";
       
       alert(alertTxt); 
       
	}
	
    //sets position of an element
    function setPosition(element,x,y,z){ 

        var pos =element.transform.position.value;
        if(x !== false) pos[0] = x;
        if(y !== false) pos[1] = y;
        if(z !== false) pos[2] = z;
        element.transform.position.setValue(pos)

    }  

    // resets anchor to top left corner
    function resetAnchor(element){ 

        element.transform.anchorPoint.setValue([0,0,0])

    }  

    //calculate wether the binary fits the video
     function checkFrames(comp){ 
            
            var fps = comp.frameRate*comp.duration;
            var bitSpace = bitWidth*bitSpacing;
            var binaryLen = fps.toString(2).split("").length;
            bitSpace = bitSpace*binaryLen;
            if(bitSpace > comp.width) {
                alert("The video is not wide enough for the bit size and spacing. Required width is "+bitSpace+"px and video is "+comp.width+"px wide.");
                return false;
             }
            return true;

     }   
 
    //create shapes
    function calcBits(comp,bitColor){ 
        
        var fps = comp.frameRate*comp.duration;
         var bin = "1"+fps.toString(2);
         var bit;
         for(var i =0; i<bin.length;i++){
             createBit(comp,bitColor,i) 
         }   
     } 
 
    //create shape
    function createBit(comp,bitColor,pos){ 
         var gfx = comp.layers.addSolid(bitColor,"bit"+pos, bitWidth, bitHeight, 1)
          resetAnchor(gfx);
         setPosition(gfx,comp.width-(bitWidth+bitSpacing)*(pos)-bitWidth,comp.height-bitHeight-bitOffsetFromBottom,false);
         //add the expression
        var expression = 'bitpos = '+pos+';frame = timeToFrames();frame = parseInt(frame,10).toString(2).split("").reverse().join("");if(frame.length < bitpos) bit = "0";else bit = frame.substr(bitpos,1);if(bit == "1") value = 100;else value = 0;';
         gfx.opacity.expression = expression;
    }    

	//run the code
	FrameNumbering(this);
    

}

Here’s the  AS3:

/*
Capture current frame from video with a barcode
Copyright Niko Helle, nikohelle.net
*/

package 
{

	import com.nhe.video.VideoPlayer;
	import com.nhe.utils.Utils;
	import com.nhe.utils.ContentLoader;
	import flash.display.Sprite;
	import flash.display.BitmapData;
	import flash.display.Bitmap;
	import flash.display.DisplayObject;
	import flash.text.TextField;
    import flash.text.TextFormat;  
    import flash.text.TextFieldAutoSize;  
	import flash.events.Event;
	import flash.geom.*;

	public class Main extends Sprite
	{


		// ignore all variables
		// set them in Main()
		// or copy them from AE's alert box after the ExtendScript is run
		
		private var _videoPlayer:VideoPlayer;
		private var _cl:ContentLoader;
		private var _blocker:Bitmap;
		private var _view:Bitmap;
		private var _bmd:BitmapData;
		private var _byteSlice:Bitmap;
		private var _tf:TextField;
		private var _lastBin:int = -1;
		private var _lastValues:Array = [];
		
		
		private var _threshold:uint = 50;
		private var _bgHeight:uint = 0;
		
		
		private var _frames = 0;
		private var _bitColor:uint = 0;
		private var _bitWidth = 0;
		private var _bitHeight = 0;
		private var _bitSpacing = 0;
		private var _bitOffsetFromBottom = 0;
		private var _bitOffsetFromTop = 0;
		
		
		
		
		public function Main() {
			
			/* Copy below parameters from the AE alert window after you have run the script*/			
			
			_bitWidth=5;
			_bitHeight=8;
			_bitSpacing=7;
			_bgHeight=14;
			_bitColor=0;
			_bitOffsetFromBottom=3;
			_bitOffsetFromTop=3;
			_frames=3600;
			
			/* copy above */
			
			// set the threshold. If the value is 50, pixel with color value of 255*50% ( = 127) or higher is decoded as "1"
			_threshold = 50;
			
			
			// calculate how many bits are used
			_frames = _frames.toString(2).length;	
			
			//create a videoplayer
			_videoPlayer = new VideoPlayer();
			_videoPlayer.addEventListener(VideoPlayer.READY_TO_PLAY,video_READY_TO_PLAY);
			_videoPlayer.addEventListener(VideoPlayer.PLAY_PROGRESS,video_PLAY_PROGRESS);

			// load the video from given path
			_videoPlayer.load("FrameNumberedVideo.flv");
			
			
			//create a contentLoader and load an image with it if you want to test with a screen capture first
			_cl = new ContentLoader();
			_cl.addEventListener(Event.COMPLETE,cl_COMPLETE);
			//_cl.load("screenshot.png");

		}
		
		// video is loaded, add trackers etc
		public function video_READY_TO_PLAY(e:Event){
			
			_videoPlayer.playVideo(1);

			// 3 ways to remove bits from view:
			// 1: addView (copies pixels from the video and never shows the actual video)
			// 2: addBlocker adds a solid colored block over the added pixels
			// 3. addMask masks the video
			
			// add video child ONLY if addView is not used!!!
			addChild(_videoPlayer);
			
			
			//addView(_videoPlayer,_bgHeight);
			//addBlocker(_videoPlayer,_bgHeight,0xffffff);
			//addMask(_videoPlayer,_bgHeight);
			
			// show the calculated binaries and frames number over the video
			addBitTracker(_videoPlayer,_bgHeight);
			
			// show the slice cut from the video that is used for detecting binaries. Slice is placed under the video
			addByteSlice(_videoPlayer);

		}
		
		// image is loaded, calculate bits and frame
		public function cl_COMPLETE(e:Event){
			
			addChild(_cl.content);
			
			//get the slice where bits are drawn and read
			
			slice(_bmd,_cl.content,_bitHeight,_bitOffsetFromBottom);
			
			//read bits
			var pixelBits = readPixels(_bmd,_frames,_bitWidth,_bitSpacing,_bitColor,_threshold);

			//covert to integer
			var frame = parseInt(String(pixelBits), 2);
			
			//trace(bin+"/"+pixelBits );
			
			// see function video_READY_TO_PLAY for explanation about function class below
			//addMask(_cl.content,_bgHeight);
			//addBlocker(_cl.content,_bgHeight,0xffffff);
			addBitTracker(_cl.content,_bgHeight);
			addByteSlice(_cl.content);
			
			// if addByteSlice is run, draw the slice
			if(_byteSlice)_byteSlice.bitmapData = _bmd;
			
			//if addBitTracker is run, show the binary and frame
			if(_tf) _tf.text = "Frame:"+frame+", binary:"+pixelBits
		}


		
		public function video_PLAY_PROGRESS(e:Event){

			//save color values in case error occurs
			_lastValues = [];
			//get the slice where bits are drawn and read
			slice(_bmd,_videoPlayer.videoLoader,_bitHeight,_bitOffsetFromBottom);
			//read bits
			var pixelBits = readPixels(_bmd,_frames,_bitWidth,_bitSpacing,_bitColor,_threshold);
			//covert to integer
			var frame = parseInt(String(pixelBits), 2);
			//trace("frame:"+frame+",pixelBits:"+pixelBits);
			// if frame is smaller that before and error has occured
			
			if(frame < _lastBin) {
				
				trace("##### current frame went backwards from "+_lastBin+" to "+frame);
				
				//stop the video so you can check the pixels and error, comment this out later
				//frame = _lastBin++; //error correction
				_videoPlayer.pause();
				for(var i=0;i<_lastValues.length;i++){
					var px = _lastValues[i];
					trace("color:"+px);
				}
			}
			else if(_lastBin > -1 && Math.abs(_lastBin-frame) != 1){
				//trace("##### current frame skipped frames from "+_lastBin+" to "+frame);
			}
			
			_lastBin = frame;
			
			// if addByteSlice is run, draw the slice
			if(_byteSlice) _byteSlice.bitmapData = _bmd;
			// if addView() is run, draw the video frame to it
			if(_view) drawView(_videoPlayer,_bgHeight);
			//if addBitTracker is run, show the binary and frame
			if(_tf) _tf.text = "Frame:"+frame+", binary:"+pixelBits

			
		}
		
		//mask out the added slice
		public function addMask(obj:DisplayObject,bgh:uint){
			var mask:Sprite = new Sprite();
			mask.graphics.beginFill(0xFF0000);
			mask.graphics.drawRect(0, 0, obj.width, obj.height-bgh);
			mask.x = obj.x;
			mask.y = obj.y;
			
			addChild(mask);
			obj.mask = mask;
			
			
		}
		
		//draw over the added slice
		public function addBlocker(obj:DisplayObject,bgh:uint,color:uint){
			_blocker = new Bitmap();
			_blocker.bitmapData = new BitmapData(obj.width, bgh,false,color);
			
			_blocker.x = obj.x;
			_blocker.y = obj.y+obj.height-bgh;
			
			addChild(_blocker);
			
			
			
		}
		
		//dont show video, copy paste pixels without the added slice
		public function addView(obj:DisplayObject,bgh:uint){
			_view = new Bitmap();
			_view.bitmapData = new BitmapData(obj.width, obj.height-bgh,false,0x000000);

			addChild(_view);

		}
		
		//get new frame as video plays on
		public function drawView(obj:DisplayObject,bgh:uint){

			var mat:Matrix = new Matrix();
			var rect = new Rectangle(0,0,obj.width,obj.height-bgh);

			_view.bitmapData.draw(obj, mat,new ColorTransform(),null,rect,false);

		}
		
		//show the frame and binary over the video
		
		public function addBitTracker(obj:DisplayObject,bgh:uint){
		
			_tf = new TextField();
			var textFormat = new TextFormat();
			textFormat.size = 10;
			textFormat.color = 0xFFFFFF;
			textFormat.font = "Arial";
			textFormat.align = "right";
			
			_tf.autoSize = TextFieldAutoSize.NONE;
            _tf.selectable = false;
			_tf.defaultTextFormat = textFormat;
			_tf.width = 200;
			_tf.height = 40;
			_tf.x = obj.width-_tf.width ;
			_tf.y = obj.height-bgh-_tf.height;
			//_tf.border = true;
			addChild(_tf);
		
		
		}
		
		// show the slice  used for binary calculation
		public function addByteSlice(obj:DisplayObject){
			_byteSlice = new Bitmap();
			addChild(_byteSlice);
			_byteSlice.x = 0;
			_byteSlice.y = obj.height;
			
		}
		
		
		// get the slice  used for binary calculation
		public function slice(bmd:BitmapData,obj:DisplayObject,bh:uint,bob:uint){
			
			if(!bmd) bmd = _bmd = new BitmapData(obj.width,1,false,0x000000);
			
			
			var mat:Matrix = new Matrix();
			
			mat.translate(0,-obj.height+bob+Math.floor(bh/2));
			var rect = new Rectangle(0,0,obj.width,1);

			bmd.draw(obj, mat,new ColorTransform(),null,rect,false);
			
			
			
			
			
			
		}

		// calculation binaries from color values
		public function readPixels(bmd:BitmapData,bits:uint,bw:uint,bs:uint,color:uint,threshold:uint):String{
				var px;
				var bin = "";
				var xpos
				
				for(var i=0; i<bits;i++){
					xpos = bmd.width - Math.ceil(bw/2)-((bw+bs)*i)
					
					px = bmd.getPixel(xpos,0);
					
					
					if(color == 0) px = (px >> 16) & 0xFF // RED
					else if(color == 1) px = (px >> 8) & 0xFF //GREEN
					else px = px & 0xFF //BLUE

					_lastValues.push(px);
					if(px >= 255*threshold/100){
						bmd.setPixel(xpos,0,0xFFFF00);
						bin = "1"+bin;
						
					}
					else {
						bin = "0"+bin;
						bmd.setPixel(xpos,0,0x0000FF);
					}

				}
				
				return bin
		
		}
		
		
		
	}	
	
}

Download AE project and Flash files (videos not included)
Download full source: videos, AE project and Flash files

The video used in the examples is © Canon. You can get the video here.

The workflow to make this work is:

– make a new AE project
– import the video
– run the AE script
– copy parameters to Flash
– render the video
– insert the file path to Flash
– Publish Flash and watch the Flash decode the frame numbers in real time
– check if Flash traces any warnings
– if there are warnings, adjust shape size and height and run the AE script again

Have fun!