Archive:MythicalLibrarian
--this page is under construction
OSX Support is in progress --Outleradam 05:29, 16 May 2010 (UTC)
The Missing Symlink
mythicalLibrarian will convert your recordings into a Video Library, free of commercials, automatically after recording. You can see an example to the right.
About The Program
mythicalLibrarian is a highly versitile tool designed to assign season and episode numbers to television shows based on show title and episode title so they are easily recognized by XBMC or Boxee. mythicalLibrarian interfaces with TheTvDb to gain information about the target file. mythicalLibrarian also has the ability to interface with MythTV's local database, XBMC's command interface, and Ubuntu GNOME desktop to provide an all-around recording recognition/managing engine which will run in the background of your back-end computer to update your library and provide status updates upon each completed job.
mythicalLibrarian pulls information supplied by your MythTV program guide for episodes, and makes a fuzzy logic match out of the supplied data. If fuzzy logic is not enough, it will match original air date. For Movies, mythicalLibrarian will rename your MythTV recordings based upon information obtained from the program guide database on your computer. If you allow commercial data to be generated for your shows, mythicalLibrarian will also copy this data with the recordings. mythicalLibrarian allows MythTV to remain in control of the files, so there is no reason not to process your recordings for boxee and XBMC.
MythicalLibrarian was designed to run under Ubuntu Karmic Koala OS through BASH 4.0
How it works
mythicalLibrarian will generate several files of it's own while running. It must have it's own working directory. To the right, you can see an example of the layout of /mythicalLibrarian, the files which it creates and their purposes. Here is the basic flow:
- You put in- 'mythicalLibrarian.sh "file/to/process.ext" "episode or movie name" "episode name"'
- Episodes- mythicalLibrarian references TheTvDb to gain SeriesID. Get current time from TheTvDb If the database is out-of-date or non existant, it will download the full series and parse it. Then, using fuzzy logic, it makes a match of the series name and gains season and episode information. If this is not possible, it will attempt a match of original airdate from your MythTV program guide database.
mythicalLibrarian will rename your recognizable episodes to: "/Show Name/Show Name.SxxExx (episode subtitle).ext"
- Movies- mythicalLibrarian references your MythTV program guide database to verify it is a movie and then obtains the year for the movie.
mythicalLibrarian will rename your movies to: "/Movie Name/movie title (Year).ext"
- Showings or no TVDB info- mythicalLibrarian will skip all processing and rename the files to "/Show Name/Show Name.S0E0 (episode subtitle).ext"
Deployment
While it is possible to manually configure the script, this may lead to configuration issues. mythicalLibrarian utilizes an advanced easy installation method which allows you to choose between using the latest stable version, or the current version located on your computer. As of right now, the SVN is updated daily, so take advantage of this easy deployment method to increase your features and functions. All users are encouraged to use the SVN version as the primary and in the event of a diagnostic failure, revert to the last stable version. Updating mythicalLibrarian to the newest version is as simple as typing mythicalLibrarian --update and then hitting enter a bunch of times.
Additional Information
- Please report problems here in the XBMC Forum thread: http://forum.xbmc.org/showthread.php?p=470402#post470402
- If you would like to see additional features and functions, additional back-end support such as tvheadend (and are willing to collect data), or additional guide data type support, please post in this thread http://forum.xbmc.org/showthread.php?t=65769
Installation Setup and Use of mythicalLibrarian
This section covers installation, setup, and use of mythicalLibrarian. After completing this section, mythicalLibrarian can run autonomously without user intervention. See the Librarian Interfaces for information on notifications and automation tasks.
Installation
This section covers installation of mythicalLibrarian.
Automatic Installation on linux
This section covers automatic installation and upgrade. Automatic upgrade will allow you to choose between downloading the newest stable, the newest SVN or continuing with the last downloaded version. I encourage SVN usage as it will assist with development.
- type the following:
sudo apt-get install curl mkdir ~/.mythicalLibrarian && mkdir ~/.mythicalLibrarian/mythicalSetup && cd ~/.mythicalLibrarian/mythicalSetup curl http://mythicallibrarian.googlecode.com/svn/trunk/mythicalSetup.sh>./mythicalSetup.sh sudo chmod +x ./mythicalSetup.sh && sudo ./mythicalSetup.sh
- mythicalSetup's first prompt will ask you to update the working copy of mythicalLibrarian or to continue with setup. If you download a new version, it will bring you back to the same prompt at which point you will select continue. This gives you the option to upgrade, or just configure the current version.
- mythicalSetup will run you through a series of prompts and perform the setup of mythicalLibrarian and all of it's features
Installation on Mac OS X
This section covers installation on Mac OS X.
Mac OSX Dependencies
This section will cover compiling binaries needed for fuzzy logic capabilities and a terminal-based menu-driven interface
- Install Xcode Tools
note: You can get Xcode from your OSX installation DVD in "Optional Installs/Xcode Tools/Xcode Tools.pkg" or you can log onto the Mac Dev center at apple.com and download it for free.
- Install X11
note: you can get X11 from your OSX installation DVD in "Optional Installs/Optional Installs.pkg" or you can build it from online sources
- Obtain and install MacPorts
note: you can get MacPorts here: http://distfiles.macports.org/MacPorts/MacPorts-1.8.2.tar.bz2 or navigate to their website here: http://www.macports.org/install.php
- Open Terminal from Finder:Applications/Utilities/Terminal
- Type the following
sudo port tre sudo port dialog
Automatic Installation on OS X
This section will cover installation of mythicalSetup, which will provide an easy interface to install mythicalLibrarian
mkdir ~/.mythicalLibrarian && mkdir ~/.mythicalLibrarian/mythicalSetup && cd~/.mythicalLibrarian/mythicalSetup curl http://mythicallibrarian.googlecode.com/svn/trunk/mythicalSetup.sh>./mythicalSetup.sh sudo chmod +x ./mythicalSetup.sh && sudo ./mythicalSetup.sh
Easy Upgrades
This section covers upgrading of mythicalLibrarian.
- type the following
sudo mythicalLibrarian --update
- mythicalLibrarian will download the newest version of mythicalSetup and perform the upgrade
Manual Installation for Linux
This section covers manual installation on Linux. Automatic installation is recommended: see below.
- Install dependencies
sudo apt-get install curl agrep libnotify-bin
dependencies breakdown:
- curl downloads web pages as text.
- agrep or tre provides fuzzy logic.
- libnotify-bin provides GNOME desktop notifications
- Download the current version of mythcalLibrarian from this download link
- Place mythicalLibrarianin /usr/local/bin
- mkdir /home/mythtv/.mythicalLibrarian
- Make mythicalLibrarian executable by typing the following:
sudo chmod +x /usr/local/bin/mythicalLibrarian
This section covers basic setup of mythicalLibrarian. To make mythicalLibrarian work, you will need to set the following settings located in text format within mythicalLibrarian.sh.You can see a picture of the settings to the right.
- MoveDir - this is your primary episode move dir.
- AlternateMoveDir- this is the folder where episodes will be moved if movedir cannot be used
- FailSafeDir- folder to place links if files cannot be moved
- PrimaryMovieDir-Folder to place movies
- AlternateMovieDir- this is where movies will be placed if PrimaryMovieDir fails
- Database username and password should be mythtv/mythtv. When installing MythTV a root user and password was assigned by the user.
Leave the rest as default, or enable and disable settings as desired.
Extra settings documentation The SYMLINK variable specifies to mythicalLibrarian how to treat the files. There are three modes to SYMLINK. MOVE is the preferred mode.
- SYMLINK=MOVE Allows XBMC to see the file and know when it has been removed from the library. When SYMLINK=MOVE and you have set up mythtv-setup to follow symlinks when deleting, MythTV's Library will function normally and XBMC will have access to all content.
- SYMLINK=LINK Keeps the original MythTV file in it's place and creates a link for XBMC to read. This link is not maintained and the symlink will remain after the original file has been deleted
- SYMLINK=Disabled Moves the file to the new location and does not provide a symlink to MythTV. On top of this, it will purge MythTV's database of all information and pictures relating to the video content. MythTV will have no memory of the recording. This mode can be used if you wish to delete files from XBMC.
Movelink.gif
Use of mythicalLibrarian on MythTV Applications
This section covers setup of MythTV without any changes to mythicalLibrarian settings. This will create only a basic setup. Make sure to follow up by checking the /home/mythtv/mythicalLibrarian/output.log file after the first run.
- Download mythicalLibrarian.sh and put it on your desktop: http://mythicallibrarian.googlecode.com/files/mythicalLibrarian.sh
- Download librarian-notify-send and put it on your desktop: http://mythicallibrarian.googlecode.com/files/librarian-notify-send
- Type the following into a terminal
sudo apt-get install curl agrep libnotify-bin sudo mkdir /home/mythtv /home/mythtv/mythicalLibrarian /home/mythtv/Episodes /home/mythtv/Movies /home/mythtv/Videos /home/mythtv/FailSafe sudo chown -R mythtv:mythtv /home/mythtv /home/mythtv/mythicalLibrarian /home/mythtv/Episodes /home/mythtv/Movies /home/mythtv/Videos /home/mythtv/FailSafe sudo mv ~/Desktop/mythicalLibrarian.sh /home/mythtv/mythicalLibrarian/mythicalLibrarian.sh sudo mv ~/Desktop/librarian-notify-send /usr/local/bin/librarian-notify-send sudo chmod +x /usr/local/bin/librarian-notify-send sudo chmod +x /home/mythtv/mythicalLibrarian/mythicalLibrarian.sh
MythTV settings
See use of mythicalLibrarian on MythTV Applications for setting up mythtv here: http://wiki.xbmc.org/index.php?title=MythicalLibrarian&action=edit§ion=4
sudo adduser *Your User Name here* mythtv
- Click System/Administration MythTV Backend setup
- Select "1. General" then select next until you get to screen 3
- Check "Follow symbolic links when deleting files".
- Select next until you get to screen 8
- Check "Allow user job #1 Jobs"
- Select next until you get to screen 10
UserJob #1 Description: mythicalLibrarian User Job #1 Command: /usr/local/bin/mythicalLibrarian "%DIR%/%FILE%"
- Select Finish to exit
- Select 6. Storage Directories
- Select Default
- Add a recording folder to which MythTV has access ie. /home/mythtv/Videos
- Add additional folders if required.
- Remove folders to which MythTV cannot run move jobs such as /lib or /var
Use of mythicalLibrarian on non-MythTV Applications
This section covers setup of non-MythTV applications. Make sure permissions are set correctly.
- edit mythicalLibrarian.sh and set Database=Disabled
- Set mythicalLibrarian up to run the job as follows:
/home/mythtv/mythicalLibrarian/mythicalLibrarian.sh "Show name" "Show Title" "File to rename"
- see Setting the Settings in this section
Using the command line
mythicalLibrarian has many command line switches. This section covers command line use of mythicalLibrarian.
mythicalLibrarian Normal Operation
This section covers normal operation of mythicalLibrarian When using mythicalLibrarian to rename files individually without database access, you can run the following command. Without database access, mythicalLibrarian can only look up episodes with title and subtitle.
mythicalLibrarian /home/mythtv/videos/1024_33020100321.mpg "spongebob squarepants" "giant squidward" mythicalLibrarian /path/to/video/file.ext "episode title" "episode subtitle"
When using mythicalLibrarian with the mythtv database access you can run mythicalLibrarian as the following command.
mythicalLibrarian /home/mythtv/videos/1024_33020100321.mpg mythicalLibrarian /path/to/mythtv/videos/filename.ext
mythicalLibrarian will integrate with the mythtv database as a user job after recordings. The mythtv user job for this is:
/usr/local/bin/mythicalLibrarian "%DIR%/%FILE%"
mythicalLibrarian --help
This section covers -h|--help
mythicalLibrarian --help
--help will display a help message containing helpful information
mythicalLibrarian --scan
This section covers -s|--scan
mythicalLibrarian --scan mpg /home/mythtv/Videos mythicalLibrarian --scan ext /path/to/video/files
--scan will search for files automatically and process them. --scan has two parmaters. The first parameter is the extension of the files to search. The seccond parameter is the folder to search.
mythicalLibrarian --doover
This section covers -d|--doover
mythicalLibrarian --doover
mythicalLibrarian keeps track of each file which fails while it runs. At any time you can type "mythicalLibrarian --doover" and all previously failed files will be processed. This process cannot be interrupted for any reason or data loss may occour.
mythicalLibrarian --undo
This section covers -u|--undo
mythicalLibrarian --undo
mythicalLibrarian keeps track of every file which has been moved. At any time you can type mythicalLibrarian --undo and all files will be reverted to their original state. There is no indication that --doover is running until it completes. This process cannot be interrupted or data loss may occour.
mythicalLibrarian --update
This section covers --update
sudo mythicalLibrarian --update
--update is a complicated process. First, mythicalLibrarian will download a copy of the mythicalSetup from the SVN repository. Next, it will download a copy of mythicalLibrarian from the SVN repository. mythicalSetup will then separate the program from the user settings. mythicalSetup perform several checks and bring up a GUI. The GUI will ask the user for preferences. mythicalSetup will attempt to set permissions and settings to appropriate defaults based upon user settings and the system. mythicalSetup will then combine the user settings with the separated program and place the file in /usr/local/bin. mythicalSetup will then call mythicalLibrarian --mythicalDiagnostics, which checks out your system. mythicalDiagnostics then calls mythcalLibrarian testfile.ext "mister" "episode 1332", which puts mythicalLibrarian into a diagnostic run. Once complete, the mythicalLibrarian in test mode, mythicalDiagnostics, mythicalSetup and update are closed and results are returned to the user.
mythicalLibrarian --mythicalDiagnostics
This section covers -m|--mythicalDiagnostics
mythcalLibrarian --mythicalDiagnostics
--mythicalDiagnostics is a step-by-step functions check of mythicalLibrarian which will test functionality. It tests for all known causes of failure. This is a good troubleshooting tool for you to work the kinks out of your settings. mythicalLibrarian runs a series of diagnostics and then runs a search for an episode of mr rodgers with the file name set to testfile.ext. testfile.ext truncates the execution of mythicalLibrarian before any file operations occur. mythicalDiagnostics will return results which should look like this
All tests complete RESULTS Installed Packages: ***PASSED*** Notifications Test: ***PASSED*** XBMC Communications: ***PASSED*** File System Checks: ***PASSED*** mythicalTesting: ***PASSED*** Overall mythicalReadiness: ***PASSED*** -----End of diagnostics-----
Notifications test and xbmc communications test will not fail overall mythicalReadiness. Only critical checks effect overall mythicalReadiness. Critical checks are dependencies, file system, and mythicalTesting.
Librarian Interfaces
mythicalLibrarian has the capability to send notifications to the Ubuntu GNOME desktop and keep your XBMC library up-to-date and clean. This section covers setup of notifications of the various platforms supported by mythicalLibrarian
XBMC Commands
This section covers setup of XBMC Commands. mythicalLibrarain can keep XBMC's library up to date, clean out deleted files and send a notification upon a completed job.
- edit mythicalLibrarian.sh in your text editor
- set XBMCUpdate, XBMCClean, and XBMCNotify to the desired settings, either Enabled, or Disabled
- Set the IP and port under XBMCIPs. Separate IP addresses with a space. eg.
XBMCIPs=( "192.168.1.110:8080 192.168.1.111:8080 XBOX:8080" )
GNOME Desktop Notifications
- mythicalLibrarian requires no-password sudo access to send GNOME notifications.
GNOME Desktop notifications can be turned on and off from within the script
GNOME Desktop notifications are used on the back-end machine by mythicalLibrarian to let you know that a file move operation is in process or an operation has completed. The external script librarian-notify-send becomes the specified username and sends a GNOME desktop notification. librarian-notify-send requires the user mythtv to have access to the sudo command without a password if run as a user job.
- Get the current version of librarian-notify-send from Google Code download link
- place the file in /usr/local/bin
- add mythtvall group
sudo addgroup mythtvall
- add mythtv to the mythtvall group
sudo adduser mythtv mythtvall
- edit the sudoers file
sudo visudo
- add the following line to the sudoers
%mythtvall ALL=(ALL) NOPASSWD:ALL
- hit ctrl-x and then y, then enter to write out and quit visudo
- make librarian-notify-send executable by typing
sudo chmod +x /usr/local/bin/librarian-notify-send
- edit mythicalLibrarian and set Notify=Enabled and NotifyUserName=*your user name*
You are done.
Troubleshooting
While mythicalLibrarian is designed to run fully autonomously, configuration issues come up from time to time. This section deals with these set-up issues
Random Errors
This section of Troubleshooting is designated for random errors which make no sense. These include errors which make you think mythicalLibrarian simply does not work.
Dependencies
This section covers dependencies of mythicalLibrarian. Usually you can find dependencies by typing "command -V" or "command --version"
- The following are the versions of the binaries which mythicalLibrarian depends upon and are not in OS Karmic Koala.
agrep -V This is agrep version 3.0, 1994. curl 7.19.5 (i486-pc-linux-gnu) libcurl/7.19.5 OpenSSL/0.9.8g zlib/1.2.3.3 libidn/1.15 dialog Version: 1.1-20080819 mysql Ver 14.14 Distrib 5.1.37, for debian-linux-gnu (i486) using EditLine wrapper notify-send 0.4.5
- The Following are the versions of the binaries which mythicalLibrarian depends upon and are included in OS Karmic Koala.
bash GNU bash, version 4.0.33(1)-release (i486-pc-linux-gnu) (GNU coreutils) 7.4 sed GNU sed version 4.2.1 grep GNU grep 2.5.4 awk GNU Awk 3.1.6 ifconfig 1.42 (2001-04-13) net-tools 1.60 replace Ver 1.4 for debian-linux-gnu at i486
Permission Errors
This section of Troubleshooting is designated for permission errors.
permission errors when run from terminal
Ensure the user running mythicalLibrarian has permission to run mythicalLibrarian. This includes read and write permissions to the working folder (default=~/mythicalLibrarian) see this post for more information: http://forum.xbmc.org/showpost.php?p=489246&postcount=36
mythicalLibrarian is not generating a log
Make sure mythicalLibrarian.sh is executable. You can set the permissions by typing "sudo chmod +x /path/to/mythicalLibrarian.sh" note: mythtv will be the user running mythicalLibrarian when run as a user job from MythTV see this post for more information: http://forum.xbmc.org/showpost.php?p=489246&postcount=36
Files are not being renamed or moved, but they are owned my mythtv
mythtv must have full access to the files. the /var/ folder does not allow users to move files without special permissions. Try moving your recording directory to your /home/ folder
mythicalLibrarian works from terminal, but not from user job
"mythtv" is the user calling the user job. If the user running mythicalLibrarian does not have access to the file, then mythicalLibrarian will not have access to the file. MythTV users can troubleshoot by changing your username with "sudo su mythtv", then running mythicalLibrarian.
mythcialLibrarian will not copy to my NFTF/FAT/EXT3 mounted drive
Additonal file systems may require editing the /etc/fstab file to change permissions. Search your OS documentation for information on how to mount a file system under a different username. Alternately, you may wish to add the user running mythicalLibrarian to your user group.
XBMC library is not updated
This section of Troubleshooting is designated for XBMC specific issues
XBMC does not show notifications
mythicalLibrarian requires the user to set the IP address of XBMC within mythicalLibrarian.sh
- ensure you have a static IP set up in your router, see your router's manufacturer's website for more information
- Open mythicalLibrarian.sh and edit the "XBMCIPs" setting to your XBMC IP.
XBMC shows notifications but Library is not updated
- Add a source http://wiki.xbmc.org/?title=Adding_Media_Sources
- Set the content of your video library http://wiki.xbmc.org/?title=Set_Contents_%28Video_Library%29
OS Speciffic issues
This section of Troubleshooting is designated for OS Specific issues
Gentoo
Gentoo OS requires upgrade of many packages. For sure you will need the package TRE. See here for version information: http://forum.xbmc.org/showpost.php?p=495697&postcount=62
Problems after upgrade to new version
This section of Troubleshooting is designated for issues after upgrade when the previous version was working
Episodes are not identifed
When episodes are not identified after an upgrade it is because the database handling has been upgraded, however, the database itself has not. Delete all database folders to correct the issue http://forum.xbmc.org/showpost.php?p=492001&postcount=40
mythicalLibrarian does not generate any files in it's working directory
Ensure the user running mythicalLibrarian has permission to run mythicalLibrarian. This includes read and write permissions to the working folder (default=~/mythicalLibrarian)
Episode recognition problems
Series title is not properly recognized
TheTvDb.com does not have the proper Zap2it id included. Edit it. The program guide uses a generic name. Apply a showtranslation. See documentation included within mythicalLibrarian.sh.
Episode title was not properly recognized
TheTvDb.com is down. Wait for TheTvDb.com to resume and then run the doover.sh file generated by mythicalLibrarian Ensure the data at TheTvDb is accurate. Correct if needed.
Gnome Desktop Notifications
This section of Troubleshooting is designated for GNOME Desktop Notifications issues
Gnome Desktop Notifications are not showing up
Ensure you have Notify=Enabled set in mythicalLibrarian.sh See this section for more information http://wiki.xbmc.org/index.php?title=MythicalLibrarian&action=submit#GNOME_Desktop_Notifications
Guide Data errors
for SchedulesDirect/Tribune/Zap2it set the GuideDataType setting in mythicalLibrarian.sh to "SchedulesDirect"
Currently Unsupported Guide data types
Support for guide data will be available with help from you. See this post: http://forum.xbmc.org/showpost.php?p=496721&postcount=90
Upgrade Issues
This section deals with issues relating to upgrading mythicalLibrarian
Notes
mythicalLibrarian is a spawn from the project BashSExx and MythSExx.
MythSExx can be used if you are strictly trying to rename episodes
mythicalLibrarian is currently in advanced beta stage.