Archive:New Video Scanner Ideas - Revision history

Karellen at 00:24, 24 August 2020

2020-08-24T00:24:50Z

← Older revision		Revision as of 00:24, 24 August 2020
Line 1:		Line 1:
	~~{{mininav\|[[Development]]}}~~
	= The source of this new concept =		= The source of this new concept =
	Many of these ideas started been discussed in these five threads in the Kodi feature suggestions forum:		Many of these ideas started been discussed in these five threads in the Kodi feature suggestions forum:
Line 49:		Line 48:
	XBMC should allow full editing of database information in order to fix any misclassifications that may occur.		XBMC should allow full editing of database information in order to fix any misclassifications that may occur.

	[[Category:Development]]		[[Category:Development-Archived]]

Karellen: Karellen moved page New Video Scanner Ideas to Archive:New Video Scanner Ideas without leaving a redirect: random notes

2020-08-24T00:24:25Z

Karellen moved page New Video Scanner Ideas to Archive:New Video Scanner Ideas without leaving a redirect: random notes

← Older revision	Revision as of 00:24, 24 August 2020
(No difference)

NedBot: web link corrections for xbmc -> kodi

2015-05-13T00:51:17Z

web link corrections for xbmc -> kodi

← Older revision		Revision as of 00:51, 13 May 2015
Line 2:		Line 2:
	= The source of this new concept =		= The source of this new concept =
	Many of these ideas started been discussed in these five threads in the Kodi feature suggestions forum:		Many of these ideas started been discussed in these five threads in the Kodi feature suggestions forum:
	* [http://forum.~~xbmc~~.~~org~~/showthread.php?t=12721 Read metadata tags from video (Movies, TV-shows, etc.) files and import into the database/library]		* [http://forum.kodi.tv/showthread.php?t=12721 Read metadata tags from video (Movies, TV-shows, etc.) files and import into the database/library]
	* [http://forum.~~xbmc~~.~~org~~/showthread.php?t=37206 Make library mode work without scrapers - import all files into the database]		* [http://forum.kodi.tv/showthread.php?t=37206 Make library mode work without scrapers - import all files into the database]
	* [http://forum.~~xbmc~~.~~org~~/showthread.php?t=30161 Filter that compare libary-mode with file-mode]		* [http://forum.kodi.tv/showthread.php?t=30161 Filter that compare libary-mode with file-mode]
	* [http://forum.~~xbmc~~.~~org~~/showthread.php?t=33710 Additional (custom) genres/sub-genres in scrapers, video database, and library filters]		* [http://forum.kodi.tv/showthread.php?t=33710 Additional (custom) genres/sub-genres in scrapers, video database, and library filters]
	* [http://forum.~~xbmc~~.~~org~~/showthread.php?t=14186 First time setup wizard]		* [http://forum.kodi.tv/showthread.php?t=14186 First time setup wizard]

	= Goals of the new scanner =		= Goals of the new scanner =

NedBot: Bot: Automated text replacement (- XBMC + Kodi ); cosmetic changes

2015-05-04T08:46:25Z

Bot: Automated text replacement (- XBMC + Kodi ); cosmetic changes

← Older revision		Revision as of 08:46, 4 May 2015
Line 1:		Line 1:

	{{mininav\|[[Development]]}}		{{mininav\|[[Development]]}}
	=The source of this new concept=		= The source of this new concept =
	Many of these ideas started been discussed in these five threads in the ~~XBMC~~ feature suggestions forum:		Many of these ideas started been discussed in these five threads in the Kodi feature suggestions forum:
	* [http://forum.xbmc.org/showthread.php?t=12721 Read metadata tags from video (Movies, TV-shows, etc.) files and import into the database/library]		* [http://forum.xbmc.org/showthread.php?t=12721 Read metadata tags from video (Movies, TV-shows, etc.) files and import into the database/library]
	* [http://forum.xbmc.org/showthread.php?t=37206 Make library mode work without scrapers - import all files into the database]		* [http://forum.xbmc.org/showthread.php?t=37206 Make library mode work without scrapers - import all files into the database]

NedBot: Robot: Substituting template: Name

2014-06-05T20:16:30Z

Robot: Substituting template: Name

← Older revision		Revision as of 20:16, 5 June 2014
Line 2:		Line 2:
	{{mininav\|[[Development]]}}		{{mininav\|[[Development]]}}
	=The source of this new concept=		=The source of this new concept=
	Many of these ideas started been discussed in these five threads in the ~~{{name}}~~ feature suggestions forum:		Many of these ideas started been discussed in these five threads in the XBMC feature suggestions forum:
	* [http://forum.xbmc.org/showthread.php?t=12721 Read metadata tags from video (Movies, TV-shows, etc.) files and import into the database/library]		* [http://forum.xbmc.org/showthread.php?t=12721 Read metadata tags from video (Movies, TV-shows, etc.) files and import into the database/library]
	* [http://forum.xbmc.org/showthread.php?t=37206 Make library mode work without scrapers - import all files into the database]		* [http://forum.xbmc.org/showthread.php?t=37206 Make library mode work without scrapers - import all files into the database]

NedBot: Bot: Automated text replacement (- XBMC + {{name}} )

2014-06-05T01:13:04Z

Bot: Automated text replacement (- XBMC + {{name}} )

← Older revision		Revision as of 01:13, 5 June 2014
Line 2:		Line 2:
	{{mininav\|[[Development]]}}		{{mininav\|[[Development]]}}
	=The source of this new concept=		=The source of this new concept=
	Many of these ideas started been discussed in these five threads in the ~~XBMC~~ feature suggestions forum:		Many of these ideas started been discussed in these five threads in the {{name}} feature suggestions forum:
	* [http://forum.xbmc.org/showthread.php?t=12721 Read metadata tags from video (Movies, TV-shows, etc.) files and import into the database/library]		* [http://forum.xbmc.org/showthread.php?t=12721 Read metadata tags from video (Movies, TV-shows, etc.) files and import into the database/library]
	* [http://forum.xbmc.org/showthread.php?t=37206 Make library mode work without scrapers - import all files into the database]		* [http://forum.xbmc.org/showthread.php?t=37206 Make library mode work without scrapers - import all files into the database]

Martijn at 08:12, 2 September 2013

2013-09-02T08:12:39Z

← Older revision		Revision as of 08:12, 2 September 2013
Line 1:		Line 1:

			{{mininav\|[[Development]]}}
	=The source of this new concept=		=The source of this new concept=
	Many of these ideas started been discussed in these five threads in the XBMC feature suggestions forum:		Many of these ideas started been discussed in these five threads in the XBMC feature suggestions forum:

>NedBot: Robot: Cosmetic changes

2013-04-27T15:24:31Z

Robot: Cosmetic changes

Ned Scott at 04:19, 12 September 2011

2011-09-12T04:19:45Z

← Older revision		Revision as of 04:19, 12 September 2011
Line 47:		Line 47:

	XBMC should allow full editing of database information in order to fix any misclassifications that may occur.		XBMC should allow full editing of database information in order to fix any misclassifications that may occur.

			[[Category:Development]]

>Gamester17: /* Checking whether a path needs scanning */

2009-01-09T14:46:30Z

Checking whether a path needs scanning

← Older revision		Revision as of 14:46, 9 January 2009
Line 27:		Line 27:
	The database will contain all the paths that have been scanned in the past, and we can use a hashing technique to detect whether the paths need rescanning. The simplest and quickest method would be to do a stat() on the path, and check the modified time. Hopefully all filesystems (on all OSes) will update the modified time of a folder if any of it's contents change (file rename, file addition, file modification, file removal etc.). We then store the path and it's hash in the db and compare to this. Alternative hashing would be to fetch the directory and hash it's contents, but this should be avoided if at all possible (smb in particular is slow at directory fetching when many files are in the folder, primarily due to the need to stat() each file).		The database will contain all the paths that have been scanned in the past, and we can use a hashing technique to detect whether the paths need rescanning. The simplest and quickest method would be to do a stat() on the path, and check the modified time. Hopefully all filesystems (on all OSes) will update the modified time of a folder if any of it's contents change (file rename, file addition, file modification, file removal etc.). We then store the path and it's hash in the db and compare to this. Alternative hashing would be to fetch the directory and hash it's contents, but this should be avoided if at all possible (smb in particular is slow at directory fetching when many files are in the folder, primarily due to the need to stat() each file).

	?Checking to see whether an item is in the library]]		=Checking to see whether an item is in the library=

	This could be improved by not only checking to see if the path to the file exists in the library, but also by hashing the file using some reasonably scheme, and storing that in the library. This will enable us to detect whether a file has moved, or whether this is simply a duplicate instance of the file. A suitable hashing algorithm may be that that is used to identify "scene releases" for automatic subtitle downloading.		This could be improved by not only checking to see if the path to the file exists in the library, but also by hashing the file using some reasonably scheme, and storing that in the library. This will enable us to detect whether a file has moved, or whether this is simply a duplicate instance of the file. A suitable hashing algorithm may be that that is used to identify "scene releases" for automatic subtitle downloading.

← Older revision		Revision as of 15:24, 27 April 2013
Line 1:		Line 1:
	=The source of this new concept=		=The source of this new concept=
	Many of these ideas started been discussed in these five threads in the XBMC feature suggestions forum:		Many of these ideas started been discussed in these five threads in the XBMC feature suggestions forum:
	*[http://forum.xbmc.org/showthread.php?t=12721 Read metadata tags from video (Movies, TV-shows, etc.) files and import into the database/library]		* [http://forum.xbmc.org/showthread.php?t=12721 Read metadata tags from video (Movies, TV-shows, etc.) files and import into the database/library]
	*[http://forum.xbmc.org/showthread.php?t=37206 Make library mode work without scrapers - import all files into the database]		* [http://forum.xbmc.org/showthread.php?t=37206 Make library mode work without scrapers - import all files into the database]
	*[http://forum.xbmc.org/showthread.php?t=30161 Filter that compare libary-mode with file-mode]		* [http://forum.xbmc.org/showthread.php?t=30161 Filter that compare libary-mode with file-mode]
	*[http://forum.xbmc.org/showthread.php?t=33710 Additional (custom) genres/sub-genres in scrapers, video database, and library filters]		* [http://forum.xbmc.org/showthread.php?t=33710 Additional (custom) genres/sub-genres in scrapers, video database, and library filters]
	*[http://forum.xbmc.org/showthread.php?t=14186 First time setup wizard]		* [http://forum.xbmc.org/showthread.php?t=14186 First time setup wizard]

	=Goals of the new scanner=		= Goals of the new scanner =
	*1. Automate as much as possible the task of getting video files into the library.		* 1. Automate as much as possible the task of getting video files into the library.
	*2. Be as user-friendly and as automated as is possible.		* 2. Be as user-friendly and as automated as is possible.
	*3. Never leave anything out of the library - anything that's there will go in, whether we can find metadata for it or not.		* 3. Never leave anything out of the library - anything that's there will go in, whether we can find metadata for it or not.
	*4. Prioritise the users data over scraped data.		* 4. Prioritise the users data over scraped data.
	*5. Reliably detect when files have moved (to save re-scraping) and when new files have been added or removed and update accordingly.		* 5. Reliably detect when files have moved (to save re-scraping) and when new files have been added or removed and update accordingly.
	*6. Never overwrite the users own data (either in nfo files or things such as playcounts or watched status).		* 6. Never overwrite the users own data (either in nfo files or things such as playcounts or watched status).

	=Implementation ideas=		= Implementation ideas =
	The scanner starts with a path, i.e. a single URL to a local filesystem. Given this the implementation should be as follows:		The scanner starts with a path, i.e. a single URL to a local filesystem. Given this the implementation should be as follows:

	For each path:		For each path:
	*1. Check it needs scanning (see below).		* 1. Check it needs scanning (see below).
	*2. If so, enumerate all video files and all folders.		* 2. If so, enumerate all video files and all folders.
	*3. For each video file in the folder, check to see whether it's in the library, and if not, obtain metadata (see below).		* 3. For each video file in the folder, check to see whether it's in the library, and if not, obtain metadata (see below).
	*4. For each subfolder in the folder, recurse in and follow this same routine.		* 4. For each subfolder in the folder, recurse in and follow this same routine.

	=Checking whether a path needs scanning=		= Checking whether a path needs scanning =
	The database will contain all the paths that have been scanned in the past, and we can use a hashing technique to detect whether the paths need rescanning. The simplest and quickest method would be to do a stat() on the path, and check the modified time. Hopefully all filesystems (on all OSes) will update the modified time of a folder if any of it's contents change (file rename, file addition, file modification, file removal etc.). We then store the path and it's hash in the db and compare to this. Alternative hashing would be to fetch the directory and hash it's contents, but this should be avoided if at all possible (smb in particular is slow at directory fetching when many files are in the folder, primarily due to the need to stat() each file).		The database will contain all the paths that have been scanned in the past, and we can use a hashing technique to detect whether the paths need rescanning. The simplest and quickest method would be to do a stat() on the path, and check the modified time. Hopefully all filesystems (on all OSes) will update the modified time of a folder if any of it's contents change (file rename, file addition, file modification, file removal etc.). We then store the path and it's hash in the db and compare to this. Alternative hashing would be to fetch the directory and hash it's contents, but this should be avoided if at all possible (smb in particular is slow at directory fetching when many files are in the folder, primarily due to the need to stat() each file).

	=Checking to see whether an item is in the library=		= Checking to see whether an item is in the library =

	This could be improved by not only checking to see if the path to the file exists in the library, but also by hashing the file using some reasonably scheme, and storing that in the library. This will enable us to detect whether a file has moved, or whether this is simply a duplicate instance of the file. A suitable hashing algorithm may be that that is used to identify "scene releases" for automatic subtitle downloading.		This could be improved by not only checking to see if the path to the file exists in the library, but also by hashing the file using some reasonably scheme, and storing that in the library. This will enable us to detect whether a file has moved, or whether this is simply a duplicate instance of the file. A suitable hashing algorithm may be that that is used to identify "scene releases" for automatic subtitle downloading.

	=Obtaining metadata=		= Obtaining metadata =
	We start with the URL for a video file. This may consist of a stack or a rar or whatever. In any case, it's treated as a single video file. From here, we must extract metadata information.		We start with the URL for a video file. This may consist of a stack or a rar or whatever. In any case, it's treated as a single video file. From here, we must extract metadata information.

	For each file that's not already in the database:		For each file that's not already in the database:
	*1. Check within the file for metadata information (tags in mp4 or mkv's, resolution/audio + video information, duration and the like).		* 1. Check within the file for metadata information (tags in mp4 or mkv's, resolution/audio + video information, duration and the like).
	*2. Check for a .nfo file containing metadata, or containing URL's to scrape and the like.		* 2. Check for a .nfo file containing metadata, or containing URL's to scrape and the like.
	*3. Generate the hash of the file and see whether this matches known online sources.		* 3. Generate the hash of the file and see whether this matches known online sources.
	*4. Scrape the filename for identifying information (eg identify tv episodes by running regexps on them, identify dvd folders and so on).		* 4. Scrape the filename for identifying information (eg identify tv episodes by running regexps on them, identify dvd folders and so on).
	*5. Use the information in 3 to classify, and if possible to do an online search (or searches) for more information.		* 5. Use the information in 3 to classify, and if possible to do an online search (or searches) for more information.
	*6. If the searches in 4 were not particularly successful (i.e. the matches were not close to the data from the filename), and if this is the only video file in this path then repeat from 3 with the parent folder included in the filename scraping procedure.		* 6. If the searches in 4 were not particularly successful (i.e. the matches were not close to the data from the filename), and if this is the only video file in this path then repeat from 3 with the parent folder included in the filename scraping procedure.
	*7. Take the best scrape thus far and check whether it's a likely match. If not, discard the information.		* 7. Take the best scrape thus far and check whether it's a likely match. If not, discard the information.
	*8. Insert all the information into the database. At minimum, the filename as title could be added.		* 8. Insert all the information into the database. At minimum, the filename as title could be added.

	Hopefully, the checks above should be enough to both identify what type of content the file contains and what the file is. It may well be that we may require additional hints from the user (eg specify content type at the root folder level) but we should try to avoid this if possible. It's better to err on the side of not fetching any metadata than misclassifying data.		Hopefully, the checks above should be enough to both identify what type of content the file contains and what the file is. It may well be that we may require additional hints from the user (eg specify content type at the root folder level) but we should try to avoid this if possible. It's better to err on the side of not fetching any metadata than misclassifying data.

	XBMC should allow full editing of database information in order to fix any misclassifications that may occur.		XBMC should allow full editing of database information in order to fix any misclassifications that may occur.

	[[Category:Development]]		[[Category:Development]]