Search sheldonbrown.com and sheldonbrown.org
You really only want to perform some video processing functions once -- for example, inverting upside-down video; image stabilization; resizing. The freeware Windows video processing application VirtualDub, (at http://virtualdub.com/) is lean and fast. By preprocessing in VirtualDub and saving the resulting file, you avoid having your main video editing suite perform the same processing again and again as it renders video.
There is another article on this site about VirtualDub. Please read it in connection with this article.
VirtualDub performs most functions using filters. One Virtualdub filter, Deshaker, is the best image stabilization software I have encountered. VirtualDub is supplied with a number of filters by default, but Deshaker is an add-on. You get it from the Deshaker Web site (at http://www.guthspot.se/video/deshaker.htm), and install it in the VirtualDub folder on your computer. Like VirtualDub, Deshaker is freeware, but the Web site requests a donation. I paid.
VirtualDub and Deshaker are available in 32-bit and 64-bit versions. The 64-vit versions run significantly faster, and so I recommend them, if your computer supports them. Some fitlers are available only in 32-bits, so you may want to keep a 32-bit version of Virtualdub as well.
Filters are the top item in VirtualDub's Video menu. When you click on that item, the filter dialog box will open. Click on "Add" to see the list of available filters. Then click on a filter in the list to add it. Filter information will appear in the filter dialog box. You may use more than one filter at a time: think of the filter list as a sort of pipeline, working in order from top to bottom of the list. So, for example, you may invert the video image, stabilize it and then adjust color, all in one pass through VirtualDub.
There is another article on this site about image stabilization in general. While excellent documentation is available on the Deshaker site, here are some specifics relating to bicycle video.
Whether you shoot it with a helmet camera or with a camera mounted on the bicycle, bicycle video is going to be shaky. There will be small, rapid shakes and also wide sweeps from a turn of the head, or of the bicycle.
Shoot at the highest resolution your camera and editing suite and tolerance for processing time can support, to avoid much loss of sharpness. I typically shoot at 1080 x 1920, and display at 720 x 1280.
Deshaker runs two passes through the video, to establish settings before actually adjusting the video. In this way, Deshaker can look ahead in time, preventing it from confusing panning with shaking. Single-pass image stabilization can run to the end of its range, resulting in jerky motion during these wide sweeps.
Deshaker lets you restrict the part of the image which it senses, and that can be useful. More about that later.
You may have Deshaker sense rotation in the first pass. You might as well, just in case you want to correct for it later.
Deshaker's default stabilization values of 1000 are designed to produce floating-on-a cloud-smooth pans. For bicycle video, I recommend lower values for horizontal and vertical pan, around 100. These settings will eliminate the rapid shake that is usual on a bicycle, while decentering the image less and producing only small border areas of the image as it shifts inside the image frame; also, minimal geometric distortion when using a camera with a fisheye lens.
On the first pass, select "Run Video Analysis Pass" in the File menu, and"Uncompressed RGB/YCbCr" in the Video menu. On this pass, Deshaker will save a log file, and VirtualDub will not save a video file. Still, processing will run much faster without a compressor -- more than twice as fast as with the XVid MPEG-4 compressor. Evidently, Virtualdub sends the video to the compressor for processing even if it isn't saving a file.
I recommend saving a different log file for each video that you stabilize. You can then re-use it with different stabilization settings on the second pass. The video must be the sameto work with the same log file.
On the second pass select "Save as AVI", and a compressor in the file menu. The free XVid MPEG-4 compressor is my favorite. Uncompressed video files are so huge (5 GB per minute!) that you will quickly run out of disk space. These files won't play at normal speed either, even from a fast hard drive.
Processing on the second pass can be slow. In 1920 x 1080 HD resolution, on my compuuter with an Intel i7-3770 processor and using the XVid MPEG-4 codec, it's about 7.5 frames per second, so it takes about 4 times as long as the clip ran. Still, I prefer to have the entire stabilized video file to work with, and so I let my computer run Deshaker when I'm working on something else. If the original file is in any format other than AVI, you can use the same filename, just with the different filetype. Keeping the original file, log file and processed file in the same folder makes them easy to find.
Deshaker itself can resize the image, and then there will be only one re-encoding. An enlarged or rotated image will be cropped unless you select a size large enough to hold all of it. Geometric changes to the video image (for example, resizing or rotation) only after the second pass will save processing time: any geometric change before Deshaker must be done on both passes, so the input on the second pass will match the log file.
Filling in borders using neighboring frames will not work correctly if you run the geometric correction before Deshaker. With a rotated image, for example, the fill will only be at the corners, where the image reaches the edges of the frame. Also, VirtualDub's rotate filter works OK to turn the image upside down or on its side, but introduces jaggies when rotating by another amount, so it's better to use another application then. It usually makes more sense, then, to run other processes after Deshaker. The exception would be if you are correcting for geometric distortions such as fisheye distortion which affect the operation of Deshaker itself.
As the image shifts position to remove shake, black border areas will appear unless you use one of the several tricks that Deshaker offers.
One is adaptive zoom, which enlarges the image to push the border areas off the screen. If adaptive zoom is turned on in Deshaker, the image will "balloon" in size whenever there is a big shake. If the camera is facing forward or rearward, this can make it look as if you are suddenly speeding up or slowing down -- even riding backwards.
Another option is to use earlier and later frames to fill in the border areas. This doesn't work perfectly with a moving camera. Images of objects closer to the camera change size faster, and so motion forward or backward only looks like zooming if everything in the picture is at the same distance from the camera. Still, the fill works well enough that it's worth doing even if the results aren't perfect. With a camera facing sideways, zooming is no help at all, but you can get a good match if the margins of the image are all at the same distance (for example, a wall).
On the second pass, setting motion smoothness for zooming to zero will eliminate the "ballooning" effect. Setting 100 for horizontal, vertical and rotational motion smoothness (or 200 at 50 or 60 frames per second) will eliminate annoying rapid shake and produce a result which is more like your perceived head motion as you are riding -- while minimizing decentering of the image.
You might also disable stabilization against rotation in the second pass if there is little rotational shake and recuding the size of border area.
If the camera doesn't change position or zoom, you can use the stabilization options more freely. Stabilized images can then look as smooth as if the camera is on a tripod. The only visible errors occur if moving objects are in different positions in the fill frames and the current frame.
If you have decided from the start to use only a fixed, zoomed-in area of your image, you can have Deshaker crop it for you. A bit of additional zoom -- about 1.1. -- will keep border areas and fill out of the picture almost all the time with the low motion-smoothness settings I recommend. If you need to change the panning and zooming to remove borders, you will probably want to do this later manually, in your editing suite. It's time-consuming, but you can make better artistic decisions than Deshaker can. This effort is mostly necessary if you want to use as much of the image as you can, even though occasional wide, rapid pans increase the size of border areas.
I recommend that you have Deshaker look backward and forward the full 30 frames to fill in the border areas. When these areas fill in seamlessly, you have a larger image area to work with. Not correcting for rotation will result in smaller but more poorly-matched border area.
Many cameras have a "rolling shutter", which scans the image from top to bottom, rather than capturing it all at once. The image will then appear stretched vertically if the camera is panning upwards quickly, and squished if the camera is panning downward. If Deshaker knows the amount of rolling shutter for your camera, it can compensate for this. It cannot, however, compensate for very rapid shake which occurs during the scan of each image. If this is vertical, objects at different heights in the image may become compressed and expanded -- the dreaded "Jello-o" effect. Horizontal panning skews background objects so vertical lines appear slanted, but this is not very noticeable. Very rapid horizontal shake can make vertical lines appear wavy.
The Deshaker Web site lists rolling-shutter values for a number of cameras. I have contributed measurements for my cameras:
The site describes a special test procedure to measure the rolling shutter amount of a camera, but you can calculate it without having to do a special test, using a video clip with a rapid pan across a building or other object that has horizontal and vertical lines at right angles. This is common enough in helmet-camera videos, when the cyclist turns the head. If the camera has a rolling shutter, the object will appear skewed. Stop the image on the screen, so you can advance it a frame at a time, just as with the test described on the Deshaker Web site. Lay a straightedge -- the edge of a sheet of paper will do -- along a vertical edge which appears skewed in the image. Extend the line upward and downward to measure the horizontal positions at the top and bottom of the screen.
Checking for rolling shutter without having to do a special test shoot.
The camera was panning and so the buildings are skewed.
The still-frame values in the rolling shutter formula serve only to make a correction if the vertical line in the test shot was not precisely vertical -- so, draw a right angle across a horizontal line in the image you are measuring from, and carry it to the top and bottom of the screen to simulate the locations of the top and bottom of a still frame. A sheet of paper conveniently has square corners and can be used to derive the vertical line from the horizontal. The horizontal and vertical lines should cross near the center of the screen, to avoid geometric errors caused by the lens itself, and especially if it is a fisheye lens. Now, more about lenses...
Deshaker can produce astonishingly good results with a conventional lens -- one which images straight lines as straight lines. Telephoto and "normal" lenses are of this type, as are some wide-angle lenses. These lenses project an accurate geometric representation of the subject onto the flat sensor in the camera. This kind of lens cannot look behind itself, or extremely far out to the sides.
A fisheye image is easy to recognize: straight lines curve like parentheses near edges of the image. A fisheye lens can cover a wider angle, up to 180 degrees and sometimes even more. Because a fisheye lens looks in very different directions at the same time and compresses the image at the edges, shake will be different there
There is a lot of image area out at the edges, and Deshaker takes an average when calculating how to reposition the image to stabilize it. Including the edges in the sensed area when using a fisheye lens will leave the center of the image shaky -- though the center is usually the main area of interest. With a fisheye lens, I recommend tracking an area only about half as high and wide as the image when running Deshaker's pass 1, the analysis pass. (For example, if you are shooting in 1080 x 1920 HD, exclude everything outside 270 pixels from the top and bottom, and 480 pixels from the right and left. If the horizon is high in your picture as it often is in a bicycle video, place the sensed rectanglecloser to the top.) Some visible shake will still occur in your final result, but it will mostly be near the edges of the image. You may want to use a smaller sensing area if you are going to crop the image significantly.
In a fisheye image with rapid panning, image stabilization will result in visible changes in image geometry as compressed and curved parts come closer to the center. The image can become "wavy" when panning direction reverses. Another way to minimize distortion is to crop the image after stabilizing it, but the best way of all is the old-fashioned way, to avoid shake as much as possible in the first place!
You may also want to exclude the margins in pass 1 even if a camera has a non-fisheye wide-angle lens (that is, if straight lines remain straight at the edges of the image) With these lenses, objects appear larger near the edges of the image, and image geometry changes noticeably as the camera pans, or if it shakes wildly. With a normal or telephoto lens, though, it's better to sense the entire image, giving Deshaker more to work with.
A feature not yet available would compensate for geometric distortions, but there is only so much software can do. Projection of an image covering angles approaching -- even exceeding -- 180 degrees onto the flat plane of a computer screen is simply not possible without geometric distortion. A wraparound image would have to be projected onto a wraparound screen to look normal. (We-re talking IMax here.)
|Articles by Sheldon Brown and others|