Hi,

Hoping someone can help. I'm trying to do motion detection on my little android, but it needs optimising. There's quite a few things I need to look out for:

1) Moving grass. Can get quite long at times!
2) Noise. It's a low quality camera
3) Dark picture. I want this to work until as late as possible (it'll turn itself off when the picture gets too dark and wait til the morning)

This is what I have so far:

http://www.mikeefranklin.co.uk/birds/birdy2.html

This is working perfectly. So long as I have ONE white pixel there, it's doing its job - and that's with each pixel 'noised' by 20% on both the previous frame and the current frame.

This is my 'optimised' java version that I tried to make:

 
package com.mikeefranklin.birdcam;
 
import android.util.Log;
 
public class Detector {
 
	int[] previous_frame;
	public Detector() {
 
	}
 
 
	public boolean detect(int[] data, int width, int height)
	{
		// greyscale first
		int min = 255;
		int max = 0;
		int i = 0;
		for(int y=0; y<height; y++){
			for(int x=0; x<width; x++){
				int avg = ((data[i] + data[i + 1] + data[i + 2]) / 3);
				data[i] = avg;
				data[i + 1] = avg;
				data[i + 2] = avg;
				min = Math.min(min, avg);
				max = Math.max(max, avg);
				i += 3;
			}
		}
 
		int[] diff_data;
		if (previous_frame != null){
			diff_data = new int[data.length];
		}else {
			diff_data = new int[0];
		}
 
		// normalize it to deal with low contrast
		int dist = Math.max(1, max - min);
		i = 0;
		boolean hasChange = false;
		for(int y=0; y<height; y++){
			for(int x=0; x<width; x++){
				int newgrey = ((data[i] - min) / dist) * 255;
				data[i] = newgrey;
				data[i + 1] = newgrey;
				data[i + 2] = newgrey;
 
				// if theres a previous frame, lets do a diff on it
				if (previous_frame != null) {
					// but first, lets morph 30% of the background in
					diff_data[i] = ((int)(0.7*(data[i]) ) + (int)(0.3*(previous_frame[i]))) - previous_frame[i];
					diff_data[i] = ( diff_data[i] < 0 ) ? -diff_data[i] : diff_data[i];
					diff_data[i] = diff_data[i] > 40 ? 255 : 0;
					diff_data[i+1] = diff_data[i+2] = diff_data[i];
					if (diff_data[i] == 255) {
						hasChange = true;
					}
				}
				i += 3;
			}
		}
 
		// save old frame, or return false here if theres no change at all yet
		previous_frame = data;
		if (previous_frame == null || !hasChange){
			return false;
		}
 
		// if there's change, we'll erode any pixels that have a black pixel within 4 of itself
		i = 0;
		int ii = 0;
		int size = 2;
		for(int y=0; y<height; y++){
			for(int x=0; x<width; x++){
				boolean erode = false;
				if (diff_data[i] == 255){
					for (int yi=-size;yi<=size;yi++){
						for (int xi=-size;xi<=size;xi++){
							int px = ((y+yi) * 3) * width + (x+xi) * 3;
							if (diff_data[px] == 0){
								erode = true;
								break;
							}
							ii += 3;
						}
					}
					// break early if there's no erosion to be had
					if (!erode){
						return true;
					}
				}
				i += 3;
			}
		}
 
		return false;
	}
}
 

Anyone got any ideas I could try?

I really do need accuracy, because my last version was resulting in way too many false positives.

I should say:

 

I've been running it on stock samsung s, so on a very old version of android. This meant I had to manually convert from YUV to RGB colors first which was causing a slow down. Just installed android 4.3 now so I'll see if I can improve it by using RGB and doing the grayscaling natively.

(sigh, it's the phone that dictates the supported image formats, not the software. grr. stuck with YUV)

Hmm. It seems I can skip greyscaling, because in YUV the Y is luminance. I can make an int array only as big as 1 per pixel now..

 

Hopefully I'll get more than 1fps once I've finished this.. :)

Hmm. Moving over to the greyscale version causes a HUGE dip in FPS.

 
 package com.mikeefranklin.birdcam;
 
import android.util.Log;
 
public class Detector {
 
	int[] previous_frame;
	public Detector() {
 
	}
 
 
	public boolean detect(byte[] data, int width, int height)
	{
 
		// between 18 and 25 fps here
 
		int min = 255;
		int max = 0;
		final int[] pixels = new int[width*height];
		for (int y = 0, yp = 0; y < height; y++) {
			for (int x = 0; x < width; x++, yp++) {
				int l = (0xff & ((int) data[yp])) - 16;
				if (l < 0) l = 0;
				pixels[yp] = (int)(l*2.55);
				min = Math.min(min, pixels[yp]);
				max = Math.max(max, pixels[yp]);
			}
		}
 
 
		// we're at 5 FPS here
 
		int[] diff_data;
		if (previous_frame != null){
			diff_data = new int[pixels.length];
		}else {
			diff_data = new int[0];
		}
 
		// normalize it to deal with low contrast
		int dist = Math.max(1, max - min);
		int i = 0;
		boolean hasChange = false;
		for(int y=0; y<height; y++){
			for(int x=0; x<width; x++, i++){
				int newgrey = ((pixels[i] - min) / dist) * 255;
				pixels[i] = newgrey;
				// if theres a previous frame, lets do a diff on it
				if (previous_frame != null) {
					// but first, lets morph 30% of the background in
					diff_data[i] = ((int)(0.7*(pixels[i]) ) + (int)(0.3*(previous_frame[i]))) - previous_frame[i];
					diff_data[i] = ( diff_data[i] < 0 ) ? -diff_data[i] : diff_data[i];
					diff_data[i] = diff_data[i] > 40 ? 255 : 0;
					if (diff_data[i] == 255) {
						hasChange = true;
					}
				}
			}
		}
 
		// down to 4 fps already
 
		// save old frame, or return false here if theres no change at all yet
		previous_frame = pixels;
		if (previous_frame == null || !hasChange){
			return false;
		}
 
		// if there's change, we'll erode any pixels that have a black pixel within 4 of itself
		i = 0;
		int size = 2;
		for(int y=0; y<height; y++){
			for(int x=0; x<width; x++,i++){
				boolean erode = false;
				if (diff_data[i] == 255){
					for (int yi=-size;yi<=size;yi++){
						for (int xi=-size;xi<=size;xi++){
							int px = (y+yi) * width + (x+xi);
							if (diff_data[px] == 0){
								erode = true;
								break;
							}
						}
					}
					// break early if there's no erosion to be had
					if (!erode){
						return true;
					}
				}
			}
		}
 
		return false;
	}
}
 
 

* this code isnt actually working yet, for some reason. :)

You have a fixed camera pointing at a fixed, raised bird table. How is grass a factor? :/

Or I guess more specifically, why does anything except the bird table get considered.

Put some LEDs on the four corners of the bird table, and when it starts processing in the morning flash them once to give the camera the boundaries within which it needs to look for motion - anything outside the lights is blanked/ignored.

(And until you get LEDs in place (or if you just don't feel like that), take a picture and do the mask manually.)

The birds always sit on the very edges, so I've had to zoom the camera out a bit to get a good enough view. This ends up with the grass always in the background. :/

I've messed with the code a bit and currently have it running at about 5fps without erosion:

 
		byte[] diff_data = data.clone();
		boolean hasChange = false;
		for (int y = 0, i = 0; y < height; y++) {
			for (int x = 0; x < width; x++, i++) {
				int l = (0xff & ((int) data[i])) - 16;
				// if theres a previous frame, lets do a diff on it
				if (previous_frame != null) {
					int pl = (0xff & ((int) previous_frame[i])) - 16;
					// but first, lets morph 30% of the background in
					int d = ((int) ((0.7 * l) + (0.3 * pl))) - pl;
					if (d < 0)
						d = -d;
					d = d > 15 ? 1 : 0;
					if (d == 1) {
						hasChange = true;
					}
				}
			}
		}
 

I could only check the pixels within a certain area, but unless I'm dealing with a square it'll be a whole load more processing. I can't really afford to do more than one loop (which is why im a bit stuck trying to work out how to do this erosion.

Ugh.

Mind you, I can certainly chop off the bottom pixels, as the bird will not be under the table..

Are you comparing 1280x960 images each time?

 

Is that the smallest the camera will go - and how expensive would resizing be?

 

Anyway, based on the current images you can easily do a simple crop - start at (equiv to) 60,350 and stop when you reach 1220,540

 

That's no extra processing, it's the boundaries to your loops (instead of 0,0 and width,height)

Reduced it down to this:

 
 
	public boolean detect(byte[] data, int width, int height) {
		for (int y = 0, i = 0; y < height; y++) {
			for (int x = 0; x < width; x++, i++) {
				int l = (0xff & ((int) data[i])) - 16;
				// if theres a previous frame, lets do a diff on it
				if (previous_frame != null) {
					int pl = (0xff & ((int) previous_frame[i])) - 16;
					previous_frame = data;
					// but first, lets morph 30% of the background in
					int d = ((int) ((0.7 * l) + (0.3 * pl))) - pl;
					if (d < 0)
						d = -d;
					if (d > 27) return true;
				}else {
					previous_frame = data;
				}
			}
		}
		return false;
	}
 

Taken the erosion off, increased the threshold. Hopefully I won't need to worry.. My test data were frames that were quite a few seconds apart, so in theory it shouldn't need to be AS precise as I was trying to make it.

On the speed front, you're casting everything to integers here?

Isn't floating point maths faster than integer maths? So maybe if you didn't cast stuff and kept it (as much as possible) as floats/decimal then it'd be quicker?

I thought ints were faster :O
Oh.

Well this is what I have so far. Not too happy with it. I'll give it a test tomorrow...

Getting about 25fps out of this when it's finding stuff (it jumps out early), and 7fps when it's not.

 

 
 
package com.mikeefranklin.birdcam;
 
import android.graphics.Bitmap;
import android.graphics.BitmapFactory;
import android.graphics.ImageFormat;
import android.graphics.Matrix;
import android.graphics.YuvImage;
import android.util.Log;
 
public class Detector {
 
	byte[] previous_frame;
 
	public Detector() {
 
	}
 
	public boolean detect(byte[] data, int width, int height) {
		for (int y = 0, i = 0; y < height; y++) {
			for (int x = 0; x < width; x++, i++) {
				int l = (0xff & ((int) data[i])) - 16;
				// if theres a previous frame, lets do a diff on it
				if (previous_frame != null) {
					int pl = (0xff & ((int) previous_frame[i])) - 16;
					int d = pl;
					d = ((int)(((0.7 * l) + (0.3 * d)))-d);
					if (d < 0){
						d = -d;
					}
					if (d > 50) {
						previous_frame = data;
						return true;
					}
				}else {
				}
			}
		}
		previous_frame = data;
		return false;
	}
}
 
 

Decided to only check a fraction of the pixels. I seem to be getting a similar result, but now it's a bit faster and I can add erosion to clear any dead pixels.

http://www.mikeefranklin.co.uk/birds/birdy2.html

Hopefully this will be enough and HOPEFULLY it'll transfer over to java well enough. In this javascript version I'm having to convert over to YUV first to simulate android so it's a bit slower.

 
fastdetect : function(that){
 
				var half_height = this.pixels.height / 2;
				var half_width = this.pixels.width / 2;
 
				for(var y=0; y<half_height; y++){
					for(var x=0; x<half_width; x++){
						var i = (y * 8) * this.pixels.width + x * 8;
 
						var r = this.pixels.data[i];
						var g = this.pixels.data[i+1];
						var b = this.pixels.data[i+2];
 
						var tr = that.pixels.data[i];
						var tg = that.pixels.data[i+1];
						var tb = that.pixels.data[i+2];
 
						var yuv = rgb_to_yuv(r, g, b);
						var yuv2 = rgb_to_yuv(tr, tg, tb);
 
						var this_col = yuv[0];
						var that_col = yuv2[0];
 
						this_col = (((5 * this_col) + (5 * that_col))/10)-that_col;
 
						if (this_col < 0) this_col = -this_col;
 
						this_col = this_col > 15 ?  255 : 0;
 
						this.pixels.data[i] = this_col;
						this.pixels.data[i+1] = this_col;
						this.pixels.data[i+2] = this_col;
 
					}
				}
				var mask = [];
				for(var y=0; y<half_height; y++){
					for(var x=0; x<half_width; x++){
						var i = (y * 8) * this.pixels.width + x * 8;
						var erode = false;
						if (this.pixels.data[i] == 255){
							for (var yi=-2;yi<=2;yi+=2){
								for (var xi=-2;xi<=2;xi+=2){
									var px = ((y+yi) * 8) * this.pixels.width + (x+xi) * 8;
									if (this.pixels.data[px] == 0){
										erode = true;
										break;
									}
								}
							}
							mask.push({'i' : i, 'erode' : erode ? 0 : 255});
						}else {
							mask.push({'i' : i, 'erode' : 0});
						}
 
					}
				}
				for (var i=0; i<mask.length; i++){
					this.pixels.data[mask[i].i] = mask[i].erode;
					this.pixels.data[mask[i].i+1] = mask[i].erode;
					this.pixels.data[mask[i].i+2] = mask[i].erode;
				}
 
return false;
			}
 

This seems to be the fastest, most accurate version yet:

 
	public boolean detect(byte[] data, int width, int height) {
 
 		// half width and height to reduce loops
		int half_height = height / 2;
		int half_width = width / 2;
 
		// boolean map to hold our threshold pic
		boolean[] map = new boolean[half_height * half_width];
		for (int y = 0, i = 0, ii = 0; y < half_height; y++) {
			for (int x = 0; x < half_width; x++, i+=2,ii++) {
				int l = (0xff & ((int) data[i])) - 16;
				if (previous_frame != null) {
					int pl = (0xff & ((int) previous_frame[i])) - 16;
					int d = pl;
					// merge 50% of each frame, lets not use division
					d = ((int)((5 * l) + (5 * d))-(d*10));
					// if theres a negative diff, lets reverse it
					if (d < 0){
						d = -d;
					}
					//build up our threshold map
					map[ii] = d > 150;
				}
			}
		}
		previous_frame = data;
		boolean erode = false;
		for (int y = 0, i = 0; y < half_height; y++) {
			for (int x = 0; x < half_width; x++, i++) {
				erode = false;
				if (map[i]) {
					loop through the pixels around it
					for (int yi=-2;yi<=2;yi++){
						for (int xi=-2;xi<=2;xi++){
							int px = (y+yi) * half_width + (x+xi);
							if (px >= 0 && px < map.length) {
								// if a pixel is eroded, carry on
								if (!map[px]) {
									erode = true;
									break;
								}
							}
						}
					}
					// if no pixels around this pixel have been eroded, we know there's probably some motion
					if (!erode) {
						return true;
					}
				}
			}
		}
		return false;
	}
 

I'm getting plenty of frames and I'm eroding noise. I might try to fit my 'normalize' code back in now, too, so I can make sure it works well at night.