How to organize your photos and videos from disparate sources (cellphones, cameras). Bash scripts.

Foreword

For the last decade we've been living in the age when affordable cloud storage services are blooming and proliferating. Hardly anyone stores impressions of their precious moments on personal computers these days. At one point the businesses realized that most people are not at all picky about the quality their images are stored in - and now there're thousands of services tailored to store and conveniently expose your pictures in galleries - Imgur, Facebook, Google Photo, you name it.

Being a dinosaur I am, I still keep my pictures and videos on a separate hard drive for multiple reasons I won't delve into right now. As the time passes and the number of pictures is nearing a hundred thousand, you start wondering if this "Ski trip" folder is any different from "Ski trip 1" folder. Then you open another folder with pics from a vacation 3 years ago, and can't remember if this particular event captured on film, as it were, happened before or after another one - just because one of them is taken with my camera and another one on my wife's, resulting in unrelated names like PIC_123 and IMG_456.

Media from camera and cellphone, taken on the same night. How do I watch it all together?

After some painstaking attempts to organize the whole bulk of media, a final approach has been crystallized. This post is to document it, and possibly help someone effortlessly group together and store pictures and videos from multiple sources like mobile phones and cameras.

Problem

The inconvenience starts to show after you and your friends/family have been to a new exciting place. Everyone is eager to snap their own pics and vids of every cool thing that catches the eye. Some have compact cameras, most people got cellphones. Now you're home, and you'd like to reminisce about your trip.
All of you snapped their fair share of pictures, so you dump all the pictures and videos to one folder:

They're all ordered wrong!

That's not a good way to see all the pictures and videos... How do you deal with this mess?

Pictures will surely contain an EXIF timestamp - that's how it's possible to determine the correct order of appearance in the final merged folder. What do we do with videos though? The ones that were filmed on mobile seem have a timestamp in their name. Videos from compact camera, as I found out experimentally, have the creation date set to the instant you started filming (provided you copied the movies as they were from SD card).

We need a way to merge all those into one place to be able to watch those in chronological order, both pictures and videos together. Now, for that we'd need some sort of unifying approach...

The Approach

• The final file name will conform to the following format: yyyy-MM-dd_HH-mm-ss_<old name>.<extension> .
• This ordering will allow for chronological order when arranging the resulting files by name, and incidentally it's the same order as ISO standard.
• The old name is optional - I keep it for identification purposes in camera-supplied media (IMG_1234 and MVI_1234), in other cases it makes the final name too long so I omit it.
• If the pictures/videos are taken in multiple time zone, they all should be shifted to the same time zone (e.g. the one you spent most time in, or your home time zone.).

That's all! Merging the files renamed like this into a single directory will result in easily browsable media library:

All those pictures and videos taken at different times and on different devices can be watched in chronological order!

To manage all those renames/timezone shifts I created some convenience Bash scripts that I'll be sharing below. They are all guaranteed to work both on Linux and Mac. Windows users have some options as well, for example I like Namexif - it can do all we need and takes up less than 2Mb of space.

Definitely try Namexif if you use Windows. It's free!

Scripts

All the scripts mentioned below can be found in my GitHub repo with shell scripts. Use the git version as I'll be updating and improving them from time to time! 

1) rename_to_exif.sh - to rename pictures

Prerequisites:
jhead and exiftool should be installed and in $PATH:
Ubuntu and Debian:
sudo apt-get install libimage-exiftool-perl jhead
Arch Linux:
pikaur -S perl-image-exiftool jhead
Mac OSX:
brew install jhead exiftool

Note: when using options 'm' and 'c', beware that the renaming is carried out with -n argument for mv command. That means that any given file can be left with its old name if renaming would cause writing over something.

How to use it: Here's a link to the script, if the github link above didn't work.
Pictures without EXIF timestamp will not be renamed when you invoke the script with just the directory. If you want to rename those as well, specify either c or m option to rename those files to creation date or modification date correspondingly:
Date source for renaming when you don't specify any option is defined in FLAG variable at the beginning of the script. You can change it if you'd like to to change the default behavior.

1.5) Shifting EXIF date (not an actual script, just one command)

What if your journey starts in Dubai and ends in Warsaw? You took a pic at 2pm just before your flight in Dubai Airport and then your phone shifts time 3 hours back and there you have it - your pictures of Poland appear before the ones that were taken back in UAE
How to use it:
Just run this command to shift the pictures' EXIF date by any number of hours, minutes, seconds or even days, months and years (okay, you won't be needing those unless your name is Doc. Emmett Brown).
For instance:
-DateTimeOriginal-='0:0:0 3:00:00' = subtract 3 hours from image's date.
-DateTimeOriginal+='0:0:0 2:00:00' = add 2 hours.
-DateTimeOriginal-='0:0:0 1:56:00' = subtract 1 hour and 56 seconds - for when you know your clock runs 4 minutes behind.

2) rename_videos.sh - to rename videos, duh

Note: same caveat as for rename_to_exif script: not only 'm' and 'c' flags, but this script in general won't rename your video if the new name is clashing with the some other one.

How to use it:
Here's a link to the script, if the github link above didn't work.
By default, this script will try to extract the timestamp from file name according to regular expression "[0-9]{8}_[0-9]{6}" and rename your video based on that timestamp. This proves useful for videos from mobile phones. Again, you have two alternatives, that is, c will make it rename videos to creation date and m to modification date:
I tried to make this script as flexible as possible to account for different timestamp formats. If this regex doesn't extract timestamps from your videos - you are free to tweak the regex and the variables pointing to where in timestamp to look for year, month and other individual portions of timestamp.

2.5) shift_vids_date - shift videos' creation or modification date

How to use it: Here's a link to the script, if the github link above didn't work.
Invoke the script without any parameters to see the full list of possible parameters.

3) rename_to_date_in_name.sh - extract datetime from file name, rename accordingly.

Note: same 'm' and 'c' flags apply. Will keep your files intact if nothing was found in name,

How to use it:
Here's a link to the script, if the github link above didn't work.
Scours a directory for media files. When it sees a specified arrangement of numbers, renames that file to yyyy-MM-dd_HH-mm-ss.

Example of full organization cycle

As an example I will merge the pictures from a trip that happened two years ago. I still have pictures and videos from this one in separate folders, which is really inconvenient:

a) 100ANDRO contains pics from an old Android phone. They will have EXIF data inside them, including timestamp. As we were using the phone to track time, I'll wager the timestamp inside those always corresponds to whichever time zone it was taken it.  Some adjustments will have to be made to consolidate all timestamps to the same time zone.

b) 114___06 - those are the pics from my compact camera. Seems like I adjusted time zone on this one only after unboxing it, and haven't changed it since then.

c) cam_vids - videos from the same compact camera. Creation (and modification) date looks like the real date it was created on - but don't forget that it could require adding/subtracting a few hours to correspond to the target time zone!

d) Just to make things interesting, we also got a folder with hand-stitched panoramas. They don't have any timestamps inside them, and we're going to get those involved later on (manually, unfortunately):

e) media - a folder that is going to contain merged files. It is now empty.

Let's actually start merging now!

1) If you haven't already, organize each device's media in it's own folder. For instance, have a separate folder for Canon camera, one more for your mobile phone and another for your friend's phone.

2) (Optional) Decide on which timezone you'd like to convert all media to - applicable if the pictures/videos were actually taken in different time zones. My preference is always the place I've spent most time at (and consequently took most pictures).

2.5) (Optional) In case you changed time zone setting on your device, separate its media files by timezones inside each device-related folder. If you can't remember if you did that or not - separate anyway - it's going to be simpler this way:

Structure you're supposed to end up with will look like so:

For future reference
Try to avoid changing the time zone setting on devices where you can afford it. Camera showing time in your home time zone is okay, but if you use your cellphone to tell time - well, keep in mind you'll have to separate pictures and videos by time zones later.
Changing time zone will cause overlaps when you fly westward. E.g. you take a pic at 5pm GMT+6, then fly west to a place with GMT+1 timezone, and it's 3pm there. Consequently, pictures taken in GMT+1 will appear earlier than the ones from GMT+6.

3) Let's start with the pictures. Rename one folder's contents to timestamps, shifting the date accordingly to convert all EXIF dates to the target time zone (if needed) :
In the example above I'm subtracting 3 hours from creation date in every video filmed in Moscow, effectively shifting those dates to GMT timezone. Then I'm renaming those in accordance with their creation date. They are ripe for merging now!

3.5) Move the files you just renamed to the final folder. See if they are in the right place in relation to the other ones. If they are not - undo the move, correct the timestamp inside them using exiftool and rename them with rename_to_exif.sh again. Repeat until this batch of photos fits snuggly in the merged folder:

TIP: Timezone setting you had on your camera and mobile phone during the trip isn't the most exciting piece of information to remember. My personal trick is to have clocks/watches on my pictures - that's how I can tell what time this picture was taken.

One example:

EXIF time on this picture is 9:05, whereas the clock at the background clearly shows 8:05.

In this case, I can clearly see that I'll have to shift the whole batch's time back one hour, then rename it to EXIF timestamp again:
Another example:

1:33pm in EXIF vs 12:33 on the clock.

Same thing as above - and actually these 2 pictures are from the same batch, confirming my initial guess about EXIF data being 1 hour ahead of time.

4) Rinse and repeat steps 3 and 3.5 until all pictures are in the media folder. What do we have left? Pictures without timestamps! Actual date and time those were taken on doesn't matter much - just invent some timestamp for those to fit right where they belong chronologically:

5) Rename the videos. Those are a bit trickier - but we still have the tools to tackle them: rename_vids.sh and shift_vids_date.sh : Done! The whole process may be easy for pics taken on just 2 devices in the same timezone or really involved for cases with multiple timezones and different devices with mixed bag of time settings - but eventually you'll end up only with this one folder containing all the media:

Inside it, all your pictures and videos will be ordered and ready to get viewed at any time:

Feature ideas for the scripts to be implemented:

• Fixed postfix string on each pic/video to identify where they came from
• Keep old name as a postfix
• Make shift_vids_date.sh support Linux

UPDATE 13 January 2019: Bonus script

Sifting through a hard drive full of old pictures, once more I found a use case that called for taking the best out of both worlds (Bash scripts in this case).
Some pictures were taken by our friends and then sent over the Internet in a way that stripped all EXIF data from them. Nonetheless, the file name indicative of the time they were taken stayed, in widespread Android format (IMG_YYYYMMDD_HHMMSS.jpg). Then I fused rename_to_exif.sh and rename_vids.sh into something that takes in a folder with pictures each having a date in specific format in its filename, and renames it accordingly (e.g. IMG_20181222_230014.jpg -> 2018-12-22_23-00-14.jpg).
If no timestamp can be found, there's usual options of either keeping the old name, or renaming to file's last creation or modification date.