Event-based camera chips are here, what’s next?(spectrum.ieee.org)
spectrum.ieee.org
Event-based camera chips are here, what’s next?
https://spectrum.ieee.org/event-based-camera-chips
31 comments
This feels much more like how biological systems operate. I wonder if neural networks trained on this kind of data would work any differently than ones trained on standard frame-by-frame camera feeds.
First paragraph -- "The pixels in event-based cameras—a technology inspired by animal vision systems…"
I was just thinking the same thing. This would would only work for biologically-plausible neuron variants, where neurons train constantly when being used. Maybe something like HTM (hierarchical temporal memory) neurons would work.
There is a stealthy startup that at least originally was doing machine vision based on the human early visual system.
You and I don’t see anything like pixels; our vision system pulls out relatively high level features like parallel lines, motion etc and starts working with that in addition to other optical data, muscle feedback on your lens shape etc. This is why many animals, including humans, freeze when frightened or when they perceive a risk: stationary objects are simply harder for most animals to pick out. The theory is (or was — I don’t know what they are doing today) is that you can see things interesting to humans and reason about them in ways interesting to humans.
Not part of that company: there are some wild but interesting theories related to this kind of thing. We pretty clearly have face-finding hardware (though not in the early visual system I presume). Do animals have this? Cats and dogs look at human faces.
I have read a theory that reading may hijack some subsystem originally used for identifying tracks (footprints).
It does appear that recognizing 2D pictures is learnt, not innate, while 1990s/2000 ML went the other way.
Some anthropomorphically-inspired design could yield systems that are more comprehensible and more useful. I wouldn't make a fetish of it (cars don't run faster than a human does) but machines should be better adapted to humans rather than the other way around (e.g. cars need special places to move around in so can only help humans when that is possible).
You and I don’t see anything like pixels; our vision system pulls out relatively high level features like parallel lines, motion etc and starts working with that in addition to other optical data, muscle feedback on your lens shape etc. This is why many animals, including humans, freeze when frightened or when they perceive a risk: stationary objects are simply harder for most animals to pick out. The theory is (or was — I don’t know what they are doing today) is that you can see things interesting to humans and reason about them in ways interesting to humans.
Not part of that company: there are some wild but interesting theories related to this kind of thing. We pretty clearly have face-finding hardware (though not in the early visual system I presume). Do animals have this? Cats and dogs look at human faces.
I have read a theory that reading may hijack some subsystem originally used for identifying tracks (footprints).
It does appear that recognizing 2D pictures is learnt, not innate, while 1990s/2000 ML went the other way.
Some anthropomorphically-inspired design could yield systems that are more comprehensible and more useful. I wouldn't make a fetish of it (cars don't run faster than a human does) but machines should be better adapted to humans rather than the other way around (e.g. cars need special places to move around in so can only help humans when that is possible).
Is the basic input still “pixel” and have we dropped these and maintain our neural network in …
Yes it seems we do not see and think in pixel, but are we really not think in pixel even though we are not aware of them ?
Yes it seems we do not see and think in pixel, but are we really not think in pixel even though we are not aware of them ?
The concept of ‘pixel’ is only about 45 years old, and wasn’t even a concept in the age of type.
If you look at the anatomy of the retina or the physiology of the early visual system you’ll see that the concept of “pixel” doesn’t even really make sense in that context.
If you look at the anatomy of the retina or the physiology of the early visual system you’ll see that the concept of “pixel” doesn’t even really make sense in that context.
Doesn't it? My understanding is basic, but as far as I know we have discrete detector cells that respond to light hitting them. We have ones that respond to all light as well as different ones that respond to only certain wavelength ranges. They aren't aligned in a matrix, sure, but that's not what makes a pixel a pixel.
We don't see the pixels because our consciousness works at a higher level of abstraction. You won't find pixel a few layers deep into a CNN either, but that doesn't mean they don't exist, the "neurons" at that layer just don't see them because they have no use for them.
That's always been my understanding at least...
We don't see the pixels because our consciousness works at a higher level of abstraction. You won't find pixel a few layers deep into a CNN either, but that doesn't mean they don't exist, the "neurons" at that layer just don't see them because they have no use for them.
That's always been my understanding at least...
I did my masters research in this field. I utilized a spiking neural network to classify images that I converted to spike trains. They are hard to train and as there was no sparse method to hold the images in the GPU the memory constraints were significant.
I once saw a presentation by Chip Elliott about a project to make video sensors ultra low power. They were redesigning the sensors from the ground up to reduce power usage in order to enable cameras to be used as very low power sensing devices. They were thinking that instead of processing a lot of captured video computation could be embedded in the sensor to make sensors that could output sophisticated signals from a dedicated chip. For example a chip that detects people or faces, a circle detecting chip, or a video flow analysis chip. All that is available in software these days, but the idea was to build it into hardware that requires almost no power. The intent was to take the huge potential of cameras as sensors and remove the high costs of power and computation as a restraint on their use, making them as cheap to use as a temperature sensor, possibly revolutionizing sensing and greatly increasing the sophistication of sensors. Combining that work with event-based cameras seems like a natural fit. Never heard any more about the project, I wonder if they've had any success. Tried a Google Scholar search but didn't find anything published.
Cool camera tech, but in regards to automotive safety uses…
>[Car makers want in-cabin monitoring of the driver to ensure they are attending to driving even when a car is autonomous mode.]
Automotive EE, I can tell you with certainty what our customers do not want. It seems pretty obvious to them that their images will not stay in the vehicle, and your car will snitch on you.
>[Car makers want in-cabin monitoring of the driver to ensure they are attending to driving even when a car is autonomous mode.]
Automotive EE, I can tell you with certainty what our customers do not want. It seems pretty obvious to them that their images will not stay in the vehicle, and your car will snitch on you.
> Automotive EE, I can tell you with certainty what our customers do not want.
It's been available in cars for 15 years now, and is used by a lot of manufacturers.
> It seems pretty obvious to them that their images will not stay in the vehicle, and your car will snitch on you.
That would be against European law:
> Driver drowsiness and attention warning and advanced driver distraction warning systems shall be designed in such a way that those systems do not continuously record nor retain any data other than what is necessary in relation to the purposes for which they were collected or otherwise processed within the closed-loop system. Furthermore, those data shall not be accessible or made available to third parties at any time and shall be immediately deleted after processing. — regulation (EU) 2019/2144
Your company will probably offer this feature soon, if they don't already do.
https://en.wikipedia.org/wiki/Driver_monitoring_system
https://www.aptiv.com/en/insights/article/what-is-a-driver-m...
It's been available in cars for 15 years now, and is used by a lot of manufacturers.
> It seems pretty obvious to them that their images will not stay in the vehicle, and your car will snitch on you.
That would be against European law:
> Driver drowsiness and attention warning and advanced driver distraction warning systems shall be designed in such a way that those systems do not continuously record nor retain any data other than what is necessary in relation to the purposes for which they were collected or otherwise processed within the closed-loop system. Furthermore, those data shall not be accessible or made available to third parties at any time and shall be immediately deleted after processing. — regulation (EU) 2019/2144
Your company will probably offer this feature soon, if they don't already do.
https://en.wikipedia.org/wiki/Driver_monitoring_system
https://www.aptiv.com/en/insights/article/what-is-a-driver-m...
I wonder if this could be used to increase the effectiveness of LIDAR in vision applications. Rather than continuously scanning the entire FOV, prioritize areas with more brightness changes. Kind of like how animal eyes have low resolution in the periphery for event detection but high resolution in the center.
If the hardware can directly tell us what pixels have changed, this should dramatically simplify how we encode and store videos, right? Ideally, the enormous amount of time computing P-frames would just not be necessary anymore.
I think video codecs do much more than just store differences between frames. They detect motion and output instructions like "move these pixels 8 to the left and 2 up". Representing that as a difference would be more data.
What is different about this compared to just grabbing frames in software and differencing these frames? It sounds to me that doing it in hardware can give a performance improvement, but only a minor one.
Probably orders of magnitude less power use
Much higher “frame rate” is one thing that comes to mind.
The event camera has no frame rate, essentially. Each photon sensor fires asynchronously. You should look into some of the applications of event cameras, they’re near magical for some things like high velocity movements and such. They have other problems though, not least of which is inaccessibility.
But wouldn't the frame rate be limited by the complexity of the scene, and the available bandwidth? What happens when buffers start to overflow?
Imagine instead of getting a grid of pixels once every 30th of a second, you instead get one pixel’s value, alone with its location, along with the time stamp at which the pixel’s change was noticed. Event cameras can have very fine time stamp resolution (orders of magnitude better than 1/30th of a second) and so a bright moving pixel can be tracked very accurately.
When you explain it like that, existing camera technology looks like the stupidest, most inefficient way to do things possible.
This seems like one of those "obvious in hindsight" discoveries, which are always the best ones.
This seems like one of those "obvious in hindsight" discoveries, which are always the best ones.
Well, digital camera technology for video is a pretty cheap extension of pre-existing digital camera technology for still photography. And capturing whole individual frames (or successive scanlines) is also a good fit for most display technologies.
It's only really in the context of computer vision and object tracking that the brute force whole-frames model starts to seem less than convenient.
It's only really in the context of computer vision and object tracking that the brute force whole-frames model starts to seem less than convenient.
And VR?
But with enough depth (bits), almost every pixel will be changing at high frequency due to small light variations.
Which probably indicates some algorithmic way the chip determines true, non-light based change.
Yet, this also means the possibility of some missed changes, of some changes being taken as lighting changes, instead of object change.
Was that a cloud, or a large shape close to the lens?
Hmm. Gonna have to read on this.
Yet, this also means the possibility of some missed changes, of some changes being taken as lighting changes, instead of object change.
Was that a cloud, or a large shape close to the lens?
Hmm. Gonna have to read on this.
> What is different about this compared to just grabbing frames in software and differencing these frames?
Normal cameras lose information by accumulating the intensity of each pixel over time until the next frame "arrives". You can not reconstruct that information anymore by diffing the frame-based output.
Event cameras on the other hand track the timestamp of the intensity change for each pixel individually. Thus they experience far less motion blur because they don't average the signal amplitude over time.
However, I haven't yet understood why their dynamic range is also better.
Normal cameras lose information by accumulating the intensity of each pixel over time until the next frame "arrives". You can not reconstruct that information anymore by diffing the frame-based output.
Event cameras on the other hand track the timestamp of the intensity change for each pixel individually. Thus they experience far less motion blur because they don't average the signal amplitude over time.
However, I haven't yet understood why their dynamic range is also better.
> However, I haven't yet understood why their dynamic range is also better.
If you’re sampling intensity changes at high frequency, then you don’t need to worry about saturating each pixel between each sample interval. Instead you would need to integrate over all the collected deltas to get an intensity value for a given time period.
If you’re sampling intensity changes at high frequency, then you don’t need to worry about saturating each pixel between each sample interval. Instead you would need to integrate over all the collected deltas to get an intensity value for a given time period.
Wondering what’s going to be the price point for these new Sony sensors. All the event cameras that we have been looking at were prohibitively expensive for most of the applications.
Here's how they can be used with drones:
https://youtu.be/BzykucxFddI
I was wondering if somebody had posted a Scaramuzza paper, guess so… lol
I think they're going to be used a lot in robotics in the future, it doesn't seem to make sense to take a bunch of frames, several million pixels each, and spend a lot of compute power to find features when consecutive frames are so similar.
https://youtu.be/6Sn9-M7qXLk?t=77